Sponsored by
Show notes
Transcript
0:00:00 Introduction
0:00:00 The most important thing about
the cloud is collaboration.
0:00:02 You can send someone a link and
you're working together right away.
0:00:05 But the downside of the cloud is that
you don't actually have the software.
0:00:09 And so what we think you should have is
a copy of the program and the data on
0:00:14 your computer, but it should still be
able to collaborate with other people.
0:00:18 Welcome to the localfirst.fm podcast.
0:00:21 I'm your host, Johanna Schickling, and I'm
a web developer, a startup founder, and
0:00:25 love the craft of software engineering.
0:00:27 For the past few years, I've been on a
journey to build a modern, high quality
0:00:31 music app using web technologies.
0:00:34 And in doing so, I've been falling down
the rabbit hole of local first software.
0:00:39 This podcast is your invitation
to join me on that journey.
0:00:42 In this inaugural episode, I'm speaking
to Peter van Hardenberg, who helped to
0:00:47 coin and popularize the term Local First.
0:00:50 As the director of the Ink & Switch
Research Lab, he's been on
0:00:53 the forefront of this work for
the better part of a decade.
0:00:57 My conversation with him today
starts with the basics of what Local
0:01:01 First is and why you, an application
developer, should care about it.
0:01:05 Before getting into it, also a big
thank you to Expo and Crab Nebula for
0:01:10 supporting this podcast and supporting
the local first ecosystem as a whole.
0:01:15 And now my interview with Peter.
0:01:19 Hello, welcome Peter.
0:01:21 Thank you so much for making it today
to kick off this new podcast with me.
0:01:26 So would you mind introducing yourself?
0:01:28 Yeah.
0:01:29 Hi, my name is Peter Van Hardenberg.
0:01:31 I am the director of
Ink & Switch Research Labs.
0:01:34 We're an independent industrial
research lab studying the future
0:01:37 of computing and tools for thought.
0:01:40 Awesome.
0:01:40 Peter and I have known
us for quite a while.
0:01:43 Back then, Peter, has been just
closing up his last chapter at Heroku
0:01:48 which I've been always a big fan of.
0:01:50 Peter found his way now working on local
first and defining what local first is.
0:01:58 Before going into it, what
has led you to local first?
0:02:02 Yeah, I mean, obviously it's
a bit of a departure, right?
0:02:04 Like, here I was building
a platform as a service.
0:02:07 You know, Heroku, for those who
don't know, is a, basically a host
0:02:12 Primarily when it started for Ruby
on Rails applications, but over
0:02:15 time for pretty much everything.
0:02:17 And the idea behind Heroku is
that you could build an app on
0:02:20 your machine and then push it.
0:02:22 To a continuous delivery system, and it
would put it live in the cloud, and there
0:02:26 are other systems like that today, you
can imagine it as you know, proto Netlify
0:02:30 or early kind of, containerized cloud.
0:02:35 Yeah, and it is a bit of a departure
to go from trying to get everybody's
0:02:38 apps running in the cloud to telling
people that maybe they should
0:02:41 just run them on their computer.
0:02:44 So I guess I should really
explain how we got there.
0:02:47 I think for me.
0:02:50 I'd like to tell a story about riding the
train in San Francisco, the subway there.
0:02:57 And my friends had been working on
a music app called Rdio which was
0:03:00 sort of a competitor to Spotify.
0:03:02 They're not around anymore.
0:03:03 And you know, as I was riding on
this train, I had been listening
0:03:08 to music in the online mode
when the train was above ground.
0:03:11 And then when we went into the tunnel,
the music would just stop working.
0:03:15 And it was really...
0:03:17 upsetting to me because like I couldn't go
back and even listen to songs I'd already
0:03:22 listened to or like scroll through the
playlists that I had because I wasn't in
0:03:27 offline mode and I just remember having
this feeling like we've really blown it
0:03:35 we've turned software from this thing that
you can just have like they call it an
0:03:39 informational good that means that like
you know the marginal cost of copying it
0:03:43 and using it as zero it's just it was this
idea that like software had gone from this
0:03:47 thing that anyone could have a copy of To
a thing that didn't even work on the train
0:03:51 because I didn't really have the app.
0:03:53 Even though I'd had the data in my
hand on my screen in front of me, in
0:03:57 my ears listening to it, somehow it had
just completely evaporated in the time
0:04:01 it took a train to go into a tunnel.
0:04:02 right.
0:04:03 And riding a train is
not this out there thing.
0:04:06 It's like this is a daily thing.
0:04:08 Yeah, is my daily commute.
0:04:09 And, you know, there are people out
there who say, Oh, yeah, you know, the
0:04:12 Internet's going to be everywhere next
week, and have you taken a subway lately?
0:04:16 You know, and like, both
in big cities, right?
0:04:20 You have this problem, but also
in more rural communities, right?
0:04:22 There's lots of places where there's
no Internet, or the Internet is slow.
0:04:26 And you can even be in places
where there is Internet, right?
0:04:28 But your cell phone reception
isn't great next to the fridge,
0:04:32 or your Wi Fi has a cut out.
0:04:33 Faced this every day when I'm just like
leaving the house and there's just this
0:04:37 little gap of the wifi stops responding.
0:04:40 There's a little bit of signal and my
apps just don't work properly anymore.
0:04:44 Yeah, everything kind of locks up.
0:04:50 The Problem with Current Software
0:04:50 And so that got me thinking,
like, well, why do we build
0:04:53 things the way we build them?
0:04:54 And the answer to some extent,
I think, was just that, you
0:04:57 know, we told people to.
0:04:59 Right.
0:04:59 We made it easy to build apps for
the cloud, and then everybody did.
0:05:03 Yeah.
0:05:03 Yeah, and so at Heroku, I had
worked on our data services mostly.
0:05:07 That was our Postgres service.
0:05:09 And I started thinking about how
really the problem is that the data
0:05:13 that you have for your application
doesn't actually live on your device.
0:05:18 It lives on the server.
0:05:20 And so I started thinking, surely
there's some way to get this data
0:05:23 from the network down to the computer.
0:05:26 And it's funny.
0:05:27 You know, Johannes, you and I
met originally through Prisma
0:05:31 and GraphQL and database schemas
and those kinds of things.
0:05:34 And this always struck me as a
really adjacent problem, which is
0:05:36 that fundamentally, on your server,
you're doing this process of taking
0:05:40 data from a database and getting
it into your sort of API back end.
0:05:44 And then you do a separate,
distinct process to get data from
0:05:47 the API to the client, right?
0:05:50 And then when it gets to the client,
you just kind of throw it away.
0:05:54 And so I started thinking like,
well, maybe it would make a lot more
0:05:58 sense if we just sort of synchronized
data between those points instead of
0:06:02 fetching it on demand all the time.
0:06:04 And in fact, that's sort of how
offline modes already worked.
0:06:06 And so I started thinking more and
more about this and got involved with
0:06:11 some great fellow researchers like
Martin Kleppmann, who's currently at
0:06:15 TU Munich, but was at Cambridge at the
time and had just written a paper about.
0:06:19 What are called JSON CRDTs, which is
like a technology for synchronizing
0:06:23 changes between two documents
in a way where you didn't just
0:06:25 have to clobber one or the other.
0:06:28 And yeah, that one thing led to
another, and we've been working on
0:06:31 the problem for about five years now.
0:06:33 I think you've been telling me
about it already many years ago.
0:06:37 And initially I wasn't quite sure
it's like, Hey, is this about only
0:06:40 offline mode, but step by step I
realize, Oh, there's much more to it.
0:06:45 And like, fundamentally this can
unlock much higher quality software
0:06:49 and also not make it harder for me
as an application developer, but
0:06:53 actually make things a lot simpler.
0:06:55 We're maybe not fully there yet.
0:06:58 But that's a vision I totally buy into.
0:07:01 So this is the path that has led you
to think what becomes local first.
0:07:07 Did you coin that term?
0:07:09 So the term local first came
around in a call we had one day.
0:07:14 I think...
0:07:15 Folks have given me credit at the lab,
but it was definitely one of those
0:07:18 open ended conversations where several
of us were kind of trying to find the
0:07:22 right term for this for a long time.
0:07:24 I'd actually planned to refer to
this as serverless technology,
0:07:28 but then Amazon came out with this
so called serverless technology,
0:07:33 which in fact was just all servers.
0:07:36 So I found that sort of like
frustrating and slightly upsetting
0:07:39 because it felt misleading.
0:07:41 But we really like the idea of calling
it local first technology because
0:07:44 the idea was to emphasize that it's
not about doing away with the cloud.
0:07:48 You know, it's not really serverless.
0:07:50 What we wanted to say was actually,
we just want to prioritize the user's
0:07:54 experience on their device and make
that easy to deliver on high quality.
0:07:58 Yeah, I like that a lot.
0:08:00 You've been mentioning the story of
Listening to Rdio on the subway and that's
0:08:05 not working, but from an application
developer perspective what would you
0:08:09 say were the frustrations that has
led you to think about local first.
0:08:14 Complexity of Modern Web Development
0:08:14 Yeah, I mean, it's a few things.
0:08:15 One is if you just think about software
development as an app developer, it's
0:08:20 sort of obscenely complicated, right?
0:08:23 You've got To build a client app,
probably using Node and JavaScript
0:08:29 and React and Vite and tailwind
and all these technologies.
0:08:35 Then you have to build like the API
backend, which is its own separate kind
0:08:39 of node app or part of your existing app
with all of its own sort of like JWTs
0:08:47 and, you know, database sync technologies
and GraphQL and, you know, you're
0:08:52 building out all that whole system.
0:08:54 And then behind that, you actually
have your data layer, which is like.
0:08:56 Postgres and so you have like in kind
of the best case in the minimum case.
0:09:02 You know, for of modern web dev,
you've got this three tier stack as
0:09:05 they call it, but the reality is,
it's not just those three tiers.
0:09:08 You also need like pager duty for
letting you know when things are down.
0:09:12 And maybe you're using like a
bunch of weird Amazon services.
0:09:15 You got a DynamoDB thing going on.
0:09:17 Then, you know, No one's hosting
things on Heroku anymore.
0:09:21 You got to put stuff up on, you know,
you need a Kubernetes cluster somewhere.
0:09:24 And, you know, it's just, it's a mess.
0:09:27 it's insanely complicated.
0:09:28 So there's just like a lot of complexity
feels gratuitous for like making an
0:09:34 app for your soccer team or something.
0:09:35 And not even from a tooling
perspective, but just the minimal
0:09:38 architectural footprint is quite
a lot for often very simple apps.
0:09:43 That's right.
0:09:44 And then on top of that,
you have to pay for it all.
0:09:46 Right.
0:09:47 You know, maybe you're a YC startup,
you get some credits to get you
0:09:50 started, but that's definitely
the drug dealer model, right?
0:09:52 Like first one's free, right?
0:09:54 And then, you know, I have personally
known a number of companies that died,
0:09:58 even though they were, uh, their product
was growing rapidly, but just the cost
0:10:05 of running their infrastructure was
too high to justify the system, right?
0:10:10 Like they didn't see how they'd be able
to turn this thing into a Like a venture
0:10:14 backed billion dollar valuation company.
0:10:16 And so they couldn't get any money
to keep the lights on despite having
0:10:20 a popular and growing product.
0:10:21 So that's crazy.
0:10:23 And then like.
0:10:24 Even on top of all of that, like, let's
say that you've managed to, you know,
0:10:28 raise enough money to hire an entire team
to be on call for you and you managed
0:10:32 to, you know, build a product that you
can convince somebody will be a billion
0:10:36 dollar, you know, valuation business at
some point and you learn all of those
0:10:40 skills and you build all those systems
out and everything, you know, ultimately,
0:10:45 it's still very fragile, right?
0:10:47 In In time, as well as in space,
you know, we've all had all of these
0:10:51 products now that we loved and used,
whether it's, you know, Google reader
0:10:56 or dark sky or, you know, pick your
favorite favorite dead tech, you know,
0:11:00 and these products go on the founders,
when they get acquired, they put up
0:11:04 this blog post about what an incredible
journey building this product has been.
0:11:08 And, you know, the incredible
journey that's happening is that
0:11:10 your software with your data is going
away and you can't have it anymore.
0:11:15 Right.
0:11:15 They're being acquired.
0:11:16 They're shutting everything down.
0:11:17 And if you're lucky, you
know, as with Rdio you can
0:11:20 download a bunch of JSON files.
0:11:22 That's all that's left
for you as the user.
0:11:23 And I just want to say, like, this is a
problem we created for ourselves, right?
0:11:28 Like, I can go and download an old
Microsoft, you know, MS DOS copy of,
0:11:32 like, WordPerfect or something from
the late 80s, and it'll still run.
0:11:38 You know, I can get doom on a floppy
disk and it'll still run, but like
0:11:42 software that came out for the cloud,
you know, it's just gone when it's gone.
0:11:46 And that's it, right?
0:11:47 This is, they call a certain period
of history, the dark ages, not, you
0:11:53 know, formally because they were bad,
but because they're dark to us today,
0:11:58 because We can't go back and look at
the records from them, they were lost.
0:12:02 And so similarly, we have built
a dark age for technology.
0:12:06 Not that it's bad, there are a lot of
things that are great, but that there
0:12:09 won't be a record of our work here, right?
0:12:12 Like the things that you and
I are building, the work that
0:12:14 I did at Heroku, the apps that
we've made and put out there.
0:12:18 Because they're not, they don't exist
outside of the one set of servers they
0:12:23 run on, you know, when they go offline,
that, that's it they're gone forever.
0:12:28 And I think that kind of sucks.
0:12:31 Yeah, I fully agree.
0:12:32 I mean, having gotten into software
development, step by step, I
0:12:36 could take my first steps with
building a little thing with HTML,
0:12:40 CSS and step by step, you want to
make it more functional, more real.
0:12:43 And then you start pulling on that
thread, like all that complexity comes
0:12:48 towards you and it's never ending.
0:12:50 So it's so much to learn, so much to
maintain, so much to pay for and operate.
0:13:00 What is Local First?
0:13:00 So maybe this is a good point
to ask, what is local first?
0:13:05 What is Local First?
0:13:06 local-first software is software
that runs on your computer and
0:13:11 collaborates with other people.
0:13:12 We think that the most important thing
about the cloud is collaboration.
0:13:16 This ubiquitous, always accessible,
copy of your data from anywhere.
0:13:20 You can send someone a link and
you're working together right away.
0:13:23 Nothing to download, nothing to install.
0:13:25 But the downside of the cloud is that
you don't actually have the software.
0:13:29 And so what we think you should have is a
copy of the program and the data on your
0:13:34 computer, but it should still be able
to collaborate with other people, right?
0:13:38 We want the good parts of the
cloud combined with the good
0:13:41 parts of the old way of building.
0:13:43 And in a way right now we have
great collaboration tools like
0:13:48 Google Drive and Figma, etc.
0:13:51 But it's all by default
lives in the cloud.
0:13:54 So you suggesting let's have the cake
and eat it too, so that the software
0:13:59 fully works on the client and we only use
the server for the absolutely necessary
0:14:05 pieces but still have collaboration.
0:14:09 That's right.
0:14:09 And Doug Engelbart, you know,
a famous computer science
0:14:14 researcher he talked about how
we can de- augment our intellect.
0:14:20 We can make things worse for ourselves.
0:14:23 And he demonstrated this really
poetically by taping a brick to a pencil
0:14:27 and asking people to write with it.
0:14:29 So I'm not here to say that there are
free lunches necessarily in solving these
0:14:33 technical problems, but I do think that
we may have taped a few giant bricks
0:14:37 to our pencils here, and that there's
a lot of unnecessary difficulty that
0:14:41 we've introduced by picking architectures
that are not well suited to the task.
0:14:47 These are in many cases,
great technologies.
0:14:49 It's just that we're deploying the
same technology that we would use to
0:14:53 build a billion person app to build a
10 person app or a million person app.
0:14:58 And the requirements are
just radically different.
0:15:01 Now, I think this sort of homogenization
of the technical stack is part of why
0:15:08 we're in the position that we're in.
0:15:10 Like Jonathan Edwards famously
said, you know, we were
0:15:13 promised bicycles for the mind.
0:15:15 But they gave us aircraft
carriers instead, right?
0:15:18 So, so we're building aircraft
carriers when we need bicycles.
0:15:23 That doesn't mean that the
aircraft carrier isn't useful.
0:15:25 It's very useful if you need to project
force in the South Pacific as the U.
0:15:31 S.
0:15:31 Navy, but it's not terribly helpful
if you want to go get groceries.
0:15:35 Right?
0:15:35 It's the wrong tool for the job.
0:15:40 How to build Local First Apps
0:15:40 So you've mentioned that it basically
should work fully on our device
0:15:44 without the server and the server
augments that software by giving
0:15:48 it additional capabilities, such as
collaboration, maybe it also helps
0:15:52 with automatic backup, et cetera.
0:15:55 So, that.
0:15:56 is very clear.
0:15:57 However, it would not be super clear
yet how I would go from typically
0:16:01 building a three tier web app to
building things in that model.
0:16:06 My intuition would be maybe
adding tons of caching, et cetera.
0:16:11 Would I still have that server database?
0:16:14 So what is the typical way
how I built local first apps?
0:16:17 Yeah, you kind of have two main problems.
0:16:19 There's the program and the data, right?
0:16:21 We'll just kind of boil it down.
0:16:23 The program in some sense is
the easy problem to solve.
0:16:26 So, you know, if you're on the web, what
you want is a PWA, progressive web app.
0:16:31 You're going to use a service
worker to cache the code.
0:16:34 You're going to try and run everything
in the browser, and you're going to
0:16:37 keep it stored so it works offline.
0:16:40 If you're on mobile, in many
cases you already build this way,
0:16:44 Yeah.
0:16:45 Like, everything that Apple
publishes first party, more or less.
0:16:49 Is already local first, right?
0:16:51 Like Apple notes.
0:16:52 It's not like you ever pull out your
phone in the elevator to, like, make a
0:16:56 note from a conversation you just had.
0:16:58 And all you see is a little spinner.
0:16:59 No, that's ridiculous, right?
0:17:01 Like Apple knows,
0:17:02 I think Apple notes is a great example
of what an app should feel like, whereas
0:17:06 I do a lot of times see that spinner on
the hallway where I don't have perfect
0:17:10 internet connectivity in other apps.
0:17:13 right?
0:17:13 Because you're at somebody's office and
you know, want to make some notes about a
0:17:15 conversation not on the WiFi or something.
0:17:18 And you're like, Oh, God, I gotta
0:17:19 wiFi password?
0:17:21 Yeah, no, that's ridiculous.
0:17:22 I'm just trying to take a note here.
0:17:24 Okay, so, so you need to get
the program on the device.
0:17:26 That's one.
0:17:27 And then you need to get
the data on the device.
0:17:29 That's too.
0:17:30 So how do you get the data on the device?
0:17:32 Well, in some very limited cases, you
just put the data into the program, right?
0:17:36 So like, I don't know, maybe
it's a reference for a board
0:17:40 game or something like that.
0:17:41 You can just download the
data and leave it in the app.
0:17:45 But of course, that's not
like the interesting case.
0:17:47 A lot of apps, most apps, I think, you
know, you have user generated content.
0:17:51 That's why you're there, right?
0:17:52 It's a map, it's a note, it's
whatever you're there to do.
0:17:55 And so in that case, what you need.
0:17:58 Is some way to get data from the cloud
to the device and then back to the cloud.
0:18:02 And the tricky part is you need to deal
with the fact that people can go offline.
0:18:07 And so that data is basically some
data I might've previously created.
0:18:12 For example, if it's the Apple
notes app, then maybe some notes
0:18:16 I've created on my Macbook.
0:18:18 I think we don't need to talk too much
about like how do you deliver the program.
0:18:23 But the data aspect that
seems really tricky.
0:18:26 So what are the problems that you've seen?
0:18:29 And what are the current best practices or
different way how we can architect things?
0:18:34 The Data Is The Hard Part
0:18:34 Yeah.
0:18:34 And again, I like anchoring
this in , how did we get here?
0:18:37 We've been downloading
data for a long time.
0:18:38 You can get a CSV file off the Internet.
0:18:40 You can email somebody
in Excel file, right?
0:18:43 We kind of get that,
like, then you have it.
0:18:45 You know, I can email you an
Excel file and then you can
0:18:48 open it like no big deal, right?
0:18:50 That seems easy.
0:18:51 Similarly, like with my with my
story about the music app, you know,
0:18:57 I could download those playlists
and just cash them on the device.
0:19:01 And so that's like a kind
of a modest thing you were
0:19:03 saying, just cache everything.
0:19:04 Cool.
0:19:05 Okay.
0:19:05 As long as all you're doing
is reading, no big deal.
0:19:09 That's fine.
0:19:10 Right?
0:19:10 That just works.
0:19:11 Cool.
0:19:11 We'll cache everything.
0:19:12 The problem is that we also still want
to be able to work when we're offline.
0:19:17 Right?
0:19:18 one our tenants is you should never, ever
be stopped from doing what you want to do.
0:19:24 By the availability or lack of
availability of another computer.
0:19:28 And I would say this is maybe also
where you can draw a line between.
0:19:32 Current, not yet local-first software
that's not entirely useless in an
0:19:36 offline scenario if it caches things
a lot, so at least I can read it.
0:19:41 But to go from there to that,
it allows you to still do
0:19:45 the job completely offline.
0:19:47 I think this is sort of like a barrier
that is incredibly hard to overcome with
0:19:52 like your more like typical three tier app
0:19:55 optimistic,
0:19:56 of caching.
0:19:57 yeah, exactly.
0:19:58 And it's not only that, but, like, it's
just complicated to think about, right?
0:20:02 Like, there's a lot of subtle problems.
0:20:04 Oh, if you know, you click the
OK and then it goes green right
0:20:07 away, but really you're running
the fetch in the back end and
0:20:11 have an
0:20:11 post request failed.
0:20:13 now we've already told
the user that it works.
0:20:16 Okay, now we need to
like raise an exception.
0:20:19 Let's make sure that we
check if we're online first.
0:20:21 Oh, but the browser
thought we were online.
0:20:23 But now it's just, it's a
nightmare as let's not do
0:20:26 I've seen many times that basically like
shows you indicates, Oh, this was great.
0:20:31 And then you see like a little spinner,
a pop up, something has gone wrong.
0:20:36 Please reload the
0:20:37 Refresh the browser.
0:20:38 You've lost all your work.
0:20:39 One of my all time least favorite
messages is on Google Docs.
0:20:44 Sometimes if you work on a document,
like while you're on a flight,
0:20:47 even though you've downloaded it,
remember to add the extension, cached
0:20:51 it before you got on the plane so
that it would be available offline.
0:20:55 And then you edit the document on
the flight and then you land and
0:20:57 you come online and it says, we're
sorry, this document has changed
0:21:01 too much while you were offline.
0:21:04 We're now resetting you to the cloud
state and then all your data is gone.
0:21:10 That's only happened to me once
or twice, but man, that hurt.
0:21:14 What we do is we can, you know, if
it's just you working again, there's
0:21:19 not really any problem here, right?
0:21:20 Like I can edit the data.
0:21:22 And then just when I come
online, we upload it.
0:21:24 The real challenge is if you have
two different things, editing
0:21:27 the data, and then you need
to merge their changes, right?
0:21:30 So like I added a track to the playlist.
0:21:34 And, you know, Spotify added
a track to the playlist.
0:21:38 Okay now we have a conflict.
0:21:40 Conflict is the technical
term we're going to use here.
0:21:42 And so that's the hard thing.
0:21:43 And lots of different systems
have approached this in different
0:21:46 ways over the years, right?
0:21:47 Like, Git gives you the ability
to merge conflicts as long as you
0:21:51 don't have edits to the same file
or the same line in the file, right?
0:21:57 CouchDB and PouchDB, you know, they
use what's called last writer wins.
0:22:01 And so it's whichever.
0:22:03 The, you know, all the systems agree
that whichever the last one was will
0:22:06 overwrite now, which one was last.
0:22:09 Okay, now we start to get into,
like, interesting distributed
0:22:11 systems questions because you can't
trust the clocks on the computers.
0:22:15 So, is it the last 1 to the server?
0:22:18 Does there have to be a
central server that decides?
0:22:20 Can we get rid of that
central server, Johannes?
0:22:22 Can we?
0:22:23 spoiler!
0:22:24 Yes, we can!
0:22:25 Sort of.
0:22:26 Sort of.
0:22:26 We'll come back to that later.
0:22:28 That makes sense.
0:22:29 If the software runs on our devices
and if the data is also in our devices,
0:22:34 now the server is kind of demoted in
a way no longer necessarily has to
0:22:40 play that absolutely central role.
0:22:43 If something goes wrong, the client
can just forget everything and get
0:22:46 everything again from the server.
0:22:48 But that's exactly the step where we
want to go beyond to have the work that
0:22:52 we do and the clients to actually trust
that and then collaborate with others.
0:22:58 that's right.
0:22:58 And I think the beauty of this model,
and we talked earlier about wanting
0:23:01 to make software simpler, right?
0:23:03 A lot of the complexity of the model
that we have today comes from having all
0:23:07 these different systems and programming
languages and Like different computers
0:23:12 involved in literally every task, right?
0:23:15 Like if I tick a box in a standard
web app, I've got my computer, which
0:23:20 then sends a request to the API
server, which then does a right to
0:23:23 the database, which then needs to
propagate that back to my computer.
0:23:26 So like, like best case theoretical.
0:23:29 We've got three systems involved.
0:23:31 Practically, though, there's also like
API gateways and like, you know, request
0:23:36 routers and like all this other stuff
is happening along the way, PG proxy
0:23:40 bouncer connection bouncers and all
this like extra hidden complexity on top
0:23:45 of the sort of notional three servers.
0:23:48 Whereas if you can move that so
that you know, yeah, you tick
0:23:51 the box and it writes it locally.
0:23:54 That's it.
0:23:54 You're done right now.
0:23:56 The problem is, well, how do I let
other people know that I took the box?
0:23:59 And so what we're trying to do is set up.
0:24:00 We call this synchronization, right?
0:24:02 So we want to synchronize the state
that you have in your local device
0:24:06 to other computers, whether that
servers in the cloud or other people
0:24:09 who you're collaborating with.
0:24:10 And so what we're doing is we're kind of
changing the relationship where instead
0:24:14 of the server mediating what's happening.
0:24:18 Right.
0:24:18 The client decides what it's doing,
and then it lets the server know,
0:24:21 like, Hey, I made this change.
0:24:22 And, you know, we can
do that incrementally.
0:24:25 And in fact, what we can do is record
all the edits that are happening on your
0:24:28 local machine and then just basically
send a log of that to the server.
0:24:32 And then similarly, the server
can send you a log of any other
0:24:36 edits that other people have made.
0:24:38 And now you have, in some sense,
a much simpler problem, which is
0:24:40 just like, okay, well, I've got all
these different sort of changes.
0:24:43 We have a change log, basically.
0:24:48 Local-first Use Cases
0:24:48 So I think there's a lot to unpack there.
0:24:51 So I'd like to understand
how can you actually do that?
0:24:55 But before going into that I'm also
wondering whether this works well for
0:25:00 some kind of applications, but not
maybe for some other applications.
0:25:04 So I can very well imagine how
this works for like Apple notes.
0:25:08 I could imagine how this works for like a.
0:25:11 More complex note taking system, maybe
all the way to the scale of Notion.
0:25:17 What I'm wondering though, whether
there's like some cutoff point where
0:25:20 this is no longer a good approach.
0:25:23 So if I imagine building something
like Facebook is that still a good fit?
0:25:28 For local-first software.
0:25:29 Is there kind of like a line in the sand
where you say, maybe theoretically still
0:25:34 possible, but absolutely not a good idea.
0:25:37 What is good fit for local-first
software and what isn't?
0:25:41 Yeah.
0:25:41 So the strength of local-first
software is when users should have
0:25:45 control over their data, right?
0:25:48 And when the devices at the edge need
to be able to operate independently,
0:25:51 local first offers a great fit.
0:25:54 Probably not a great fit for
an ATM, aTM just deciding when
0:25:58 it's offline to spit out money.
0:26:00 You know, without any way to let
other nodes know what it's doing.
0:26:03 When the users have a lot of agency and
authorship over the data, local-first
0:26:07 software makes a ton of sense.
0:26:08 If you're trying to manage an external
resource, like cash in a bank account
0:26:12 or a meeting room booking or something
like that, then it makes less sense.
0:26:16 I think social media is an
interesting middle ground.
0:26:20 I think there's a lot of benefits.
0:26:23 to giving users control over their
data and an agency over their data.
0:26:27 But there's interesting
scale and indexing problems.
0:26:30 So I can imagine, in fact,
there are social networks
0:26:33 that are totally local first.
0:26:35 A great example of one
is called scuttlebutt
0:26:37 and that's a local first social network.
0:26:39 But you know, it comes with a
bunch of architecture trade offs.
0:26:42 It's designed for offline use.
0:26:44 And so that means when you
join scuttlebutt, or at least
0:26:46 When I joined Scuttlebutt, you
had to download like 10 gigs
0:26:50 Wow
0:26:51 sailboats and like, you know, social
media posts that you will never
0:26:55 read just to be able to get sort
of bootstrapped into the system.
0:27:00 don't think that's a necessary
consequence of local for software.
0:27:03 design that they chose to prioritize,
which is like you should be able
0:27:07 to have the whole enchilada.
0:27:09 But yeah, I think again, right,
like, I guess another useful thing
0:27:13 to consider is like how often will
users engage with the same data?
0:27:19 So, like, Wikipedia, probably not a
great local first app in some sense.
0:27:24 Though I'm told Patrick Collison
built an offline Wikipedia
0:27:27 as one of his first ventures.
0:27:29 so, you know, I guess
there's a use for it.
0:27:31 But, you know, if you only go to an
article once and then you read it and
0:27:35 then you never need it again, and that's
like a random access pattern and we
0:27:38 can't know what you're going to want
to read, then it's probably not a great
0:27:41 fit for local-first software, right?
0:27:42 Like, you're not editing, you're
only reading, and you're not
0:27:44 going back to the same data.
0:27:46 On the other hand, like, a recipe book
kind of the inverse of that, which
0:27:51 is like, kitchens often have a lot
of, like, metal and RF interference.
0:27:55 Transcribed So even though you're
fetching data, you might want to
0:27:57 have the recipe, you want to be able
back say, what did I make last year?
0:28:01 might be your favorite recipe
and you might have a little note
0:28:04 that you've scribbled on it.
0:28:06 absolutely, scribbling in the
margins of your recipe books
0:28:08 is a time honored tradition in,
like, every household, I think.
0:28:12 And so, right, like, this is where
it comes back to that question,
0:28:16 which is, like, is a user taking data
for themself and then they can have
0:28:19 authorship over it going forward?
0:28:21 It's probably a good fit.
0:28:22 the more kind of like trust, like
control and trust, I think are
0:28:27 related here and the less you can
trust all the participants in your
0:28:32 system, the harder it probably will
be to build in this local first way.
0:28:37 Since you need to mediate what
is typically referred to as auth
0:28:41 and your typical three tier app.
0:28:44 This needs to be thought of
probably quite differently when
0:28:48 you build local first apps.
0:28:50 I think this is a bigger
conversation for another day.
0:28:54 I believe that local-first
software will provide a much higher
0:29:02 standard of privacy, security and
authentication than a cloud service can,
0:29:10 I think there's a whole bunch of new
best practices that we need to figure out
0:29:15 that might not be fully discovered yet.
0:29:17 yeah, it's definitely an open
area of work that I'm not here to
0:29:20 tell you that problem is solved.
0:29:22 I'm here to tell you that, like,
the potential, I think, is huge
0:29:25 for this, but the reality is
we're a long way from having,
0:29:28 And that's perfect.
0:29:29 that's what I want to help with this
podcast give an outlet to discuss
0:29:33 those sort of particular areas,
such as what does authentication
0:29:38 look like in a local first way?
0:29:40 And I think something that
makes the local first world
0:29:42 already so rich are those like.
0:29:44 Early products that are being built in
this way where we can share and learn from
0:29:49 their experiences and their experiments.
0:29:52 So I'm really looking forward to
deep dives on those various topics.
0:29:57 tons of people doing great work.
0:29:58 Of course, blue skies out there
doing it in more of what you'd call a
0:30:01 federated way, but they're definitely
thinking about these problems.
0:30:03 And there's lots of
people building products.
0:30:07 Data Syncing
0:30:07 So going a step back again.
0:30:10 When you say synchronization, I'm
thinking the Dropbox icon back in the
0:30:15 days when I was using Dropbox a lot.
0:30:18 So that does synchronization.
0:30:19 I think a lot of people are
still familiar with that.
0:30:23 Is that kind of like the mental
model, how I think about it?
0:30:26 Git also has a notion of.
0:30:29 Synchronizing, if a bit more explicit
with push and pull how would I do that
0:30:35 for my data where I think about my
data, maybe more in terms of SQL tables.
0:30:41 So how do I translate that very
intuitive model of syncing to the
0:30:47 more harsh realities of my app,
0:30:50 This is a great flashlight to shine on
like how the world has changed, right?
0:30:55 So if you go and look at
Dropbox, what does Dropbox sync?
0:31:00 It syncs files.
0:31:02 What is a git repo on your machine?
0:31:05 A git repo is a bunch of files.
0:31:08 And then git and Dropbox both do something
special behind the scenes to get those
0:31:14 files other places or from other places.
0:31:16 Okay.
0:31:16 What is a Google Doc?
0:31:18 You have a Google Doc on your
computer, you don't actually have it.
0:31:22 You're just looking at it, right?
0:31:24 And that's why Google can throw away
all your data because there have
0:31:27 been too many edits by other people.
0:31:29 Nevermind spending three
hours on a flight, rewriting
0:31:32 the stupid blog post, right?
0:31:33 Cause it's, you didn't,
I didn't have the doc.
0:31:36 I had a cache.
0:31:37 I had a copy.
0:31:37 I had a view.
0:31:39 And because somebody else had edited
it, the actual doc, the place had
0:31:44 changed and mine was now trash.
0:31:47 . Google Docs are places.
0:31:49 And I think this is like a really
interesting kind of duality, right?
0:31:53 Because we want both
Properties for our documents.
0:31:56 I want to be able to send a
link to someone so that they
0:31:58 can edit a document with me.
0:32:00 Right?
0:32:00 That feels like a place.
0:32:02 But I also want to be able to save
a copy and email it around or
0:32:05 have it on my computer and look
at it and know that it's mine.
0:32:08 Right?
0:32:08 So we want properties of both
this kind of like object like
0:32:13 experience that we get from files.
0:32:15 And the place like experience
we get from cloud documents.
0:32:18 And I think that's where we get
into like a real fundamental
0:32:20 tension between the two.
0:32:22 Existing Local First Products
0:32:22 Do you feel any sort of software that
we use on a daily basis is getting
0:32:27 the closest to what you think could
be a good resolution of the tension?
0:32:33 I think there are lots of companies
that have made incremental progress.
0:32:40 Just to kind of, you know,
speak to some exemplars.
0:32:42 I mean, obviously, I think the basic idea
behind an app generally, I think it's
0:32:48 kind of problematic, which is it's taking
your data and putting it into their app.
0:32:52 So you can't access somewhere,
access it anywhere else.
0:32:55 Great example, again, is Apple Notes.
0:32:58 Plain text notes, more or less, but you
can't open them in VS code or notepad or
0:33:04 anything else or Dropbox notes, right?
0:33:06 Like, we've managed to turn text notes
into closed proprietary formats that are
0:33:12 specific to a single piece of software.
0:33:14 That's sort of a bummer.
0:33:15 You can't sync your
Apple Notes with Dropbox.
0:33:18 But okay, I still think though
that, like, within this world
0:33:21 of files versus You know, things
versus places, files versus links.
0:33:26 Apple notes is like a
nice middle ground, right?
0:33:28 Like you can make a note, it always
works offline, but you can share
0:33:31 them with other people and see the
edits and get updates from people.
0:33:33 So that's really cool.
0:33:35 I think Figma is doing really
interesting things with like,
0:33:38 branches and decentralized workflows.
0:33:41 If you ever talk to people who work in
a large org that uses Figma, You know,
0:33:46 there's this problem, which is that
like when you're working on a design in
0:33:50 the early stages, you don't want other
people to look at your work, right?
0:33:55 You're they don't want someone hopping
in and going, I don't like that color.
0:33:58 It's like, yeah, I'm not done.
0:34:01 back later, right?
0:34:02 You want that creative
privacy where you can explore.
0:34:04 And so people have found all these
creative ways to build that in Figma.
0:34:08 And I think there's somewhat
cognizant of that organizationally.
0:34:11 So I think that's like
a good example of yeah.
0:34:14 Like trying to plumb some of those
depths, I think, though, for the most
0:34:18 part, I mean, you know, we could give
a bit of credit to Google Docs, right?
0:34:21 Like, yes, it requires a browser
extension and like literally half the
0:34:25 time that you at least I get on a flight.
0:34:27 I realized that although I had the browser
extension plug installed, it was in
0:34:31 the wrong profile, or I didn't remember
to cash the document or whatever, but
0:34:34 you can at least get some semblance of
that offline support from Google Docs.
0:34:40 And of course, you know, Git is an
exemplar as well in terms of, like,
0:34:43 Git plus GitHub gives you something.
0:34:46 It's clunky.
0:34:47 I'm not here to tell you Git's a great
user experience, but, you know, Git
0:34:51 gives you this ability to work kind
of in both worlds where you can share
0:34:54 links to files through GitHub, but you
can have a full copy of the repository
0:34:58 with its history on your machine.
0:35:00 Right.
0:35:01 Yes.
0:35:01 I think there's a lot to be explored
from a user experience perspective,
0:35:06 like patterns that fit the domain
of the software patterns that
0:35:10 users intuitively can work with.
0:35:13 But I think in terms of how we actually
building that there's probably still
0:35:17 a long way to go and think it's
super, super fortunate that there are
0:35:23 companies already pushing forward.
0:35:25 there whether it being Figma or Google
with Google Docs or even Notion who I
0:35:31 think is like not super sophisticated
in terms of syncing is yet a good
0:35:36 example of like how far you can get
while building collaborative software
0:35:41 that is not super fine granular in
terms of its syncing resolution.
0:35:51 Conflict Free Replicated Data Type (CRDTs)
0:35:51 And then when you actually get to the
implementation, when you're saying
0:35:55 syncing I've already seen the term CRDT
quite a bit associated with local first.
0:36:04 Can you briefly touch on what CRDTs are?
0:36:07 We probably don't have enough
time to fully go into it.
0:36:10 Maybe you can,
0:36:11 let's give a very high level primer.
0:36:13 I think this is a topic that
needs a deeper dive, but
0:36:16 we'll give the quick intro.
0:36:18 So the idea behind the CRDT is
that it stands for a conflict
0:36:22 free replicated data type.
0:36:28 data type is pretty easy.
0:36:30 There are lots of different CRDTs.
0:36:31 You can have a set, you can have
an array, you can have a counter.
0:36:35 We work on one that's
basically like a JSON file.
0:36:38 Right?
0:36:38 So it has sets and arrays and
counters and maps and things
0:36:41 and when you say data type, this is now
a thing that I'm using to build my app.
0:36:45 So in the same way, if I build a
JavaScript app and I would use a map
0:36:50 or set an array, now I would use one of
those CRDTs as foundation for my data.
0:36:57 Why are you doing that?
0:36:58 Well, because it's replicated.
0:37:00 What does means?
0:37:01 It means you can take that data
type and you can send it to a
0:37:03 bunch of different computers.
0:37:05 All right, cool.
0:37:06 typical JavaScript map can't do that.
0:37:09 Yeah, I mean, you can call toJSON
and then post it somewhere.
0:37:12 And that leads us to the last part of
the description, which is conflict free.
0:37:17 And that's where the heavy work gets done.
0:37:20 So replicating your data is
not that hard as long as you
0:37:23 don't care what happens to it.
0:37:24 You could just, I don't know, open
your window and start shouting,
0:37:27 you know, byte codes at the window.
0:37:29 Technically, you're replicating it,
I guess, if anybody's listening.
0:37:32 But you know, the conflict free
part, that's where it gets tricky.
0:37:35 How do you make sure that if two
people both edit the document, One,
0:37:41 that you can merge the changes, but
two, that nothing's lost, right?
0:37:47 Like, it's really easy to merge data if
you just say, well, I'm going to throw,
0:37:51 always throw away your data and keep mine.
0:37:54 But that's not a very
useful strategy, right?
0:37:56 So the whole idea behind CRDTs
is lots of different approaches.
0:38:00 Tons of different implementations.
0:38:02 It's a very active area of research
and there's lots of great ideas
0:38:06 still unfolding out there right now.
0:38:08 And it's really simple conceptually
though, which is like, if two people
0:38:12 edit a thing, you should be able to
merge their edits together and both
0:38:16 get the same answer without having to
go to recourse to a central authority.
0:38:21 That seems like a very interesting
area that I want to learn more about.
0:38:26 When thinking about CRDTs, there are a few
technologies that come to mind, whether
0:38:30 that's being Yjs by Kevin Jans, who's been
working on this already for many years,
0:38:35 then you all been working on auto merge
. So I would love to learn a lot more
0:38:40 about those, but right now we'll shelve
it away and say, okay, CRDTs is a great
0:38:45 strategy if you need to implement syncing.
0:38:48 Is that
0:38:48 Yeah, it is, and I think there's a
couple of, I'm not a big believer
0:38:53 in silver bullets or kind of like
magical thinking with technology.
0:38:56 There's a set of tradeoffs here, right?
0:38:58 Like, what happens when
you have a conflict in git?
0:39:01 Well, you're busting open your text editor
and looking for a bunch of less than signs
0:39:06 in a row and hoping you found them all
before you push the stupid commit again.
0:39:10 You know, similarly, CRDTs don't
generally have that problem, but
0:39:15 you can still have conflicts, right?
0:39:16 Like if you and I both edit the title of
a document, You know, the CRDT isn't going
0:39:21 to be able to know which one is right.
0:39:24 So there's interesting, just like,
challenges here that go beyond,
0:39:28 abstract computer science problems.
0:39:31 This is a really fundamental problem.
0:39:33 The computer can't know what
the title should be, right?
0:39:36 And in the central cloud model, what
happens is, like, one of us will find
0:39:40 out when we go to write that we lost.
0:39:42 And so you know the
problem happens right away.
0:39:45 And so what makes CRDTs interesting from
like a user experience design problem is
0:39:50 that you can have conflicts discovered
a long way away from where they're
0:39:56 created, either in space or in time.
0:39:59 And so that actually has serious
user experience consequences.
0:40:03 This can also happen to you and get right.
0:40:05 Like all of us developers
listening here know what it's like.
0:40:09 To, like, try and merge a long
running branch and realize that
0:40:11 somebody else has, like, completely
rewritten a file you edited and
0:40:14 you're like, oh no, what do I do?
0:40:17 Do I just throw my entire week
of working away and start over?
0:40:20 Or do I, like,
0:40:21 There's no free lunch here.
0:40:22 So it's semantically diverges a
lot or changes a lot, then the
0:40:26 user still needs to help out.
0:40:28 And similarly, the CRDT can't actually
recognize all the conflicts, right?
0:40:32 The technical conflicts we can catch.
0:40:36 But again, if you sort of think
about software, Git is kind of
0:40:40 like a bad CRDT in some ways.
0:40:42 Even if we edit different files,
that doesn't mean the tests will
0:40:46 pass or the software will be useful.
0:40:48 I may have renamed a
function that you were using.
0:40:51 Both can commit cleanly, but then
when we try and run the tests, it
0:40:54 won't even parse and it crashes.
0:40:56 We can think about all those things, and
I think that's the, design side of the
0:41:00 problem, and then, there's lots of neat
stuff to talk about on the implementation
0:41:03 side, but we can on from that for now.
0:41:05 We'll back to those later.
0:41:06 Yeah, so going beyond syncing and
CRDTs, obviously that's a very
0:41:11 important aspect, syncing in general
for building local-first software,
0:41:15 as it's one of the pillars, like the
software should work on my device,
0:41:20 and we should be able to collaborate.
0:41:22 So if we come back to the.
0:41:25 The Challenges of Building Local First Apps
0:41:25 Which sort of problems does a
developer who's typically rather
0:41:29 building that three tier web app,
what sort of new problems does the
0:41:34 developer now need to think about or
maybe think about differently aside
0:41:38 from syncing if you want to build
your next app in a local first way.
0:41:42 The biggest one is architectural.
0:41:44 When you grow up in this
environment of you know, the
0:41:48 database is the source of truth.
0:41:49 And then things sort of roll out, you
know, now you have different problems.
0:41:53 Like, how do I make sure that I have the
data, you know, when my user is offline?
0:41:57 Can I make sure that I've fetched
everything they need so that
0:42:00 they know before they go offline?
0:42:02 Like, yes, you're up to date.
0:42:04 Similarly, you know, if you
can be partially online, right?
0:42:09 Like, maybe you're online and
you can talk to the sync server.
0:42:12 But somebody else can't.
0:42:15 So, you know, how can you know that
you're seeing what other people
0:42:18 are seeing that you're seeing the
things somebody else has sent, right?
0:42:21 I like to give examples from the
field when we talk about this stuff.
0:42:24 Facebook Messenger has like a really
simple version of this problem, you
0:42:29 can still reply to a message when
you're offline with your phone, but
0:42:31 you don't get the little checkbox
that says, Yep, that's been sent.
0:42:35 And it's not until somebody else
takes the phone out and looks at it
0:42:38 and the little, you know, miniature
avatar icon drops below the message
0:42:42 that you know that they've seen it.
0:42:45 What does that mean for source code?
0:42:49 Or like a text document.
0:42:51 I mean, we can't just move the avatar
to the bottom of the page, right?
0:42:54 So there's like really interesting
challenges here around kind of,
0:42:57 thinking a little differently about
how you build your app, right?
0:43:01 And similarly, just because you
have the data locally or could
0:43:04 have the data locally doesn't mean
that it's going to load quickly.
0:43:08 If you store everything as like a
long log of edits in an indexed DB,
0:43:12 it could take you longer to load than
if you were fetching it from the web.
0:43:15 So we need to be very
thoughtful about these things.
0:43:17 But the result of Using this approach
has one really killer benefit that I
0:43:23 just still delight in every day, which
is that, you know, there's the old thing
0:43:28 about, you know, it ran on my machine.
0:43:30 Well, we'll ship your machine.
0:43:31 And that's how Docker was born, right?
0:43:35 Like, it's sort of like that, though,
with local-first software, like, if
0:43:41 The Benefits of Local First for Developers
0:43:41 You have the whole thing.
0:43:43 So you're done.
0:43:44 Right?
0:43:45 Like, as long as you have a sync
server somewhere, you're finished.
0:43:48 And actually, maybe this is a good moment.
0:43:49 Can I talk a little bit
about peer to peer for a sec?
0:43:51 Please.
0:43:52 Yes.
0:43:53 So, I think one misconception I hear
from people is like, Oh, local first.
0:43:57 That means you're anti cloud
and you don't like servers.
0:44:01 Not at all.
0:44:02 Like, peer to peer technologies are
interesting and there have been You
0:44:09 know, use cases where they were really
vital to achieving users aims, like, you
0:44:14 know, when you want to download an entire
album full of Linux ISOs, you know, then
0:44:21 BitTorrent is, like, really good, and
you can see why having central servers
0:44:24 would be a problem for that, but in the
case of most software, you know, one,
0:44:31 peer to peer systems actually don't work
very well in a lot of network conditions.
0:44:35 You can, you know, you can run BitTorrent
on your home network with a little
0:44:39 bit of work or a VPN, but at the coffee
shop or places like that, those peer
0:44:44 to peer systems really struggle because
the network tends to be configured in
0:44:46 a way that doesn't allow them to work.
0:44:48 But more than that, right?
0:44:50 Like, if you and I are collaborating
on a peer to peer system, right?
0:44:54 Like, I'm on a laptop most of the time.
0:44:58 I don't want my work to disappear
completely just because I close the lid,
0:45:03 We want servers.
0:45:04 We want them
0:45:07 mean, they literally should serve us.
0:45:10 They right, not us serving them, right?
0:45:12 We're trying to make the server
serve us rather than vice versa.
0:45:17 Love it.
0:45:18 That's great.
0:45:19 And so, yeah, the big difference here
in this model is that you still want
0:45:23 servers, but their responsibility is just.
0:45:26 to host data and distribute it.
0:45:29 And so, you know, yeah, you want it.
0:45:31 It needs to be online for things
to be successful because peer to
0:45:34 peer systems don't really work.
0:45:36 I have spent many years trying to
like adopt peer to peer architecture
0:45:41 before kind of finally giving in
and embracing this perspective.
0:45:45 And then even if they do work, they
don't solve the problem, which is
0:45:47 that in fact, you want your data
to be online even when you're not
0:45:51 Right.
0:45:52 You know,
0:45:52 there's an interesting role it can play.
0:45:54 I know a few folks who haven't
given up yet on peer to peer.
0:45:59 And maybe in the cases where it does work,
it can be an optional part of the process.
0:46:03 Of a technology stack, let's say
you're in a smart home setting where you
0:46:07 can assumptions about software works.
0:46:11 Well, peer to peer software is great and
can work really well, but it, I have not
0:46:19 seen a design or implementation of peer
to peer systems that are like really
0:46:26 robust in all network environments and
where you don't just actually have a few
0:46:33 centralized servers hiding off stage.
0:46:36 While you're saying peer to peer,
but actually, you know, 20 percent of
0:46:39 traffic is going to some server like
a great example here is WebRTC, right?
0:46:44 If you're familiar with WebRTC,
people think of it as like this
0:46:46 peer to peer system built into the
browser, but it's not for two reasons.
0:46:50 First.
0:46:51 While it is true that WebRTC allows two
computers to talk directly to each other,
0:46:56 in order for them to find out about
each other to have that conversation,
0:47:00 you need a server to introduce them.
0:47:03 The second is that, and opinions vary
about the prevalence of this, but I have
0:47:09 seen credible reports that are sort of
on the order of 20 percent of WebRTC
0:47:14 connections, cannot be made peer to peer.
0:47:18 And are instead sent through what's
called a turn server, which is just relay
0:47:22 that just bounces the traffic for you.
0:47:25 And like, that's not to say WebRTC is bad.
0:47:28 I mean, it's super useful.
0:47:29 We use it every day in our,
like, video conferencing lives.
0:47:34 But that, it's not
really like peer to peer.
0:47:37 It's like client server with a peer
to peer fast path when it's available.
0:47:42 I think there's lots
to be explored as well.
0:47:44 That's more like on the networking side.
0:47:47 And I think depending on for some
use cases, like . Let's say you
0:47:51 build software for a warehouse or
maybe a cruise ship or something.
0:47:55 I think there, there might be
real use cases, and I'm really
0:47:58 you have control...
0:47:59 are exploring it.
0:48:01 Yeah, if you have control over the
network, you totally can and actually
0:48:05 our friends at ditto where former lab
collaborator Ray McKelvey works have
0:48:10 built a really great business building
local-first software using a different
0:48:14 kind of CRDT than the one we use for
airplanes and like fast food restaurants.
0:48:20 They have like real scaled out
production, industrial deployments
0:48:24 of this technology in the field.
0:48:29 The Seven Ideals of Local First
0:48:29 So if you Google Local First right now,
you might come across the blog post
0:48:34 and essay that you've been a co author
of where you first introduce the ideas
0:48:39 of Local First, and as part of it, you
have those seven ideals of Local First.
0:48:45 So I think this would be enough to,
fill an entire episode on to really
0:48:48 do each ideal justice, but maybe
you can just Summarize those seven
0:48:54 ideals and what they mean for you.
0:48:57 Yeah.
0:48:57 Well, we've talked a bunch already
about how building local-first
0:49:01 software is good for developers, right?
0:49:02 Like, your distribution costs are better.
0:49:05 Your on call experience is better.
0:49:06 Your software is less
complicated, hopefully.
0:49:09 But the original motivation behind
developing the ideas of local first was
0:49:13 really focused on kind of user experience
and not in the sort of what corner
0:49:18 radius should the buttons be or what shade
of green for the call to action, but like
0:49:23 the real core experience that the user has
a person who's consuming your software.
0:49:28 And so we identified with Martin
Kleppmann and Adam Wiggins and
0:49:32 Mark McGranagan seven ideals.
0:49:35 That we think local-first software
should aspire to the first one is no
0:49:39 spinners right next frame or your money
back like you should never go to open
0:49:47 something and just see like a spinner
because your network connection is
0:49:50 bad or whatever else like that's just
it's inexcusable we have the fastest
0:49:54 computers in history with the biggest
drives ever in the most bandwidth like
0:49:58 it's just ludicrous that our own personal
data would not be available to us.
0:50:03 Number two your work is
not trapped on one device.
0:50:06 No one wants to be in a world where like
you dropped your phone, over a cliff
0:50:11 or off a ledge or whatever and suddenly
your data is all gone with it, right?
0:50:14 You left your laptop on the train
and now you've lost your life's work.
0:50:18 No, we don't want that.
0:50:19 So your work should be able to
synchronize across all your devices
0:50:21 and be available on every device.
0:50:24 Simultaneously with this idea
of not having spinners, right?
0:50:26 The idea is we want all seven of
these things to be true together.
0:50:30 Number three is that
the network is optional.
0:50:32 As we've discussed, like, you often find
yourself in places where the network
0:50:36 either is not available because of
technical reasons, or just because,
0:50:41 you know, you want to be offline.
0:50:43 Right.
0:50:43 There's something to be said for
making that choice, turning off
0:50:45 the Wi Fi signal, staying focused,
and still being able to work.
0:50:49 We still insist on seamless
collaboration with your colleagues.
0:50:53 That's number four, right?
0:50:56 It's, if it's not collaborative,
it's local only, not local first.
0:51:00 Another one that I think is
really important is this notion,
0:51:04 number five, the long now.
0:51:07 I can get on a single micro SD card.
0:51:13 Like, every game that was ever
released for Super Nintendo.
0:51:16 And those will work a
hundred years from now.
0:51:20 Like, how much of the software
that you've built in your career
0:51:23 will work a hundred years from now?
0:51:25 How much of it will still be
around ten years from now?
0:51:28 For many of us, so much of
our work just evaporates.
0:51:31 And it's not just for the vanity
of wanting to see your work survive.
0:51:37 When you're talking to
people, writing books, or...
0:51:41 You know, who have built their
careers around like a long term
0:51:44 project, you know, it could take 10
years to do the research for a book.
0:51:48 How many cloud services that were
around 10 years ago are still around?
0:51:52 Or how many of that are around
today will be around in 10 years.
0:51:55 You know, you can't control who's
going to get acquired or shut down.
0:51:59 So I think that's a big part of it.
0:52:01 Number six is security
and privacy by default.
0:52:04 You know, why is the default that when
I use software, the product managers at
0:52:09 the company or the support people can
just read my data whenever they want?
0:52:13 I mean, they don't.
0:52:15 I hope they don't.
0:52:16 They most...
0:52:17 Places say they won't,
0:52:19 Right.
0:52:20 but you know, they, they could, right.
0:52:23 That's pretty messed up.
0:52:25 And then number seven, ultimately,
you know, we want the users to
0:52:29 retain ownership and control.
0:52:31 Right.
0:52:32 And that's in some sense, kind of
a distillation and restating of
0:52:35 some of the other points, but like.
0:52:38 You know, if someone can take it
away from you, it's not yours.
0:52:41 We want you to be able to have
the software to make decisions
0:52:44 about when you update it.
0:52:45 If you update it, who you share
your data with, when you share
0:52:49 your data with them and to put
those tools into the user's hands.
0:52:52 We want them to have that feeling.
0:52:55 It's a genuine fact of having the thing
and having agency and ownership over.
0:53:03 those are the seven ideals.
0:53:04 I think it dovetails with a lot
of the developer benefits, right?
0:53:08 Like, sort of the same underlying
decisions that make things
0:53:11 easier for developers also
can improve things for users.
0:53:15 But those are the seven ideals.
0:53:17 I would say from a product perspective
and from the end user perspective, I think
0:53:21 there's nothing controversial about it.
0:53:23 I think like almost all developers who
build apps for the cloud today would
0:53:28 probably still say, yes, those would
be great characteristics for our apps.
0:53:33 It's just hard to build apps like
in a cloud first way that have
0:53:38 all of those characteristics.
0:53:39 So I think this is why we need a bit of
a different model, how to build software.
0:53:44 And, those from a user
perspective are amazing.
0:53:47 And from a developer
perspective, everything becoming
0:53:50 simpler sounds incredible.
0:53:53 How ready is Local First?
0:53:53 Zooming out a bit, I love all the ideas
of Local First, and so this is not the
0:53:59 topic of today's show like, I'm also,
for the last two years, working on a
0:54:05 Local First app myself, but, Pretending
I'd be two years earlier and just trying
0:54:11 to make sense of what is local first?
0:54:14 Is that for me?
0:54:15 How do I go about it?
0:54:17 Let's say I'm at the beginning of
my journey and I'd be asking you as
0:54:21 a fellow builder for advice, like,
Hey, Is local first ready for me
0:54:25 to build my next app with it today?
0:54:28 And if so what does my
typical path look like?
0:54:31 Is there like a Ruby on
Rails for it already?
0:54:34 Is this a good fit for me?
0:54:36 Do I rather have to be like
an adventurous explorer?
0:54:39 Or can I already go into
this with a mindset of what
0:54:42 is the most popular stack?
0:54:44 And that's also what I got to adopt here,
0:54:47 So we're recording this in 2023.
0:54:49 And today I would say building
a local first web app is likely
0:54:55 to require inventing something
at some point along the way.
0:55:00 Like this is not like just pushing
stuff to Netlify and following
0:55:04 posts off stack overflow.
0:55:05 This is a new thing and you're going
to have to figure some things out
0:55:09 and I hope someday that this is as
easy as falling off a log, right?
0:55:13 Like at Heroku, we used to always say
make the right thing the easy thing and
0:55:18 make the easy thing the right thing.
0:55:20 And so if we can do that, if we can get
there together as a community, then, you
0:55:24 know, Hopefully when people find this
podcast a few years from now, they're
0:55:28 like, wow, we've come a long way.
0:55:30 But today I think, you know,
what I would recommend is like,
0:55:33 you can build a react app.
0:55:36 You can use a CRDT like, Yjs or
auto merge are probably the most
0:55:42 mainstream choices right now.
0:55:44 But there are some technologies that I can
0:55:46 Oh yeah.
0:55:47 And there's a Vulkan.
0:55:48 io.
0:55:49 There's ElectricSQL.
0:55:50 There's like a ton of people
coming out and trying things.
0:55:54 And depending on what looks good to you
you should try one and see how far you
0:55:59 So there's basically an ecosystem of
tooling already for those use cases.
0:56:03 there's always a bit of like, it depends.
0:56:05 So one might better for your
use case than another.
0:56:08 But think what saying is like, I need
to invent maybe something along the way.
0:56:13 Maybe I need to do end to end
encryption, or maybe I need to do
0:56:17 permission control in a certain way.
0:56:20 Where none of the available
tools yet do exactly what I need.
0:56:23 And this is where I need
to get my own hands dirty,
0:56:26 Yeah, you'll probably need to
figure out how to deal with
0:56:30 some of those offline use cases.
0:56:32 You're going to have to decide
who you want to give this data to.
0:56:34 I think that's one that
everybody kind of wants different
0:56:37 things when we talk to people.
0:56:39 And I think eventually
some standards will emerge.
0:56:41 There'll be probably
two or three patterns.
0:56:42 Most people adopt, but at the
moment, because everybody's asking
0:56:45 for different things, people
are all making different things.
0:56:49 But I think that's a that sort of ACLs and
access permissions is an area you should
0:56:53 expect to spend some time thinking about.
0:56:55 But like, I gotta say, it feels
amazing to just stand up an app.
0:57:02 In a short matter of time pointed at some
kind of a sync server and then just be,
0:57:08 you know, working on it with other people.
0:57:10 And I hope everybody will take a
take the time to at least have that
0:57:15 experience to try building something
small, whether it's with automerge or Y.
0:57:18 J.
0:57:18 S.
0:57:18 or ElectricSQL or anything else.
0:57:21 And that experience, it's just.
0:57:25 It's so eye opening for me as somebody who
spent all these years building these big
0:57:30 complicated web apps for the cloud to just
say actually, it can be pretty simple.
0:57:37 And when it is simple, it feels great.
0:57:40 You feel like you have like, like wings.
0:57:43 You're just able to move so fast,
you're able to get so much done.
0:57:46 And it's like someone's taking
the brick off the pencil.
0:57:48 You just, like, you don't even realize
how much hassle you're putting up
0:57:52 with every day until it goes away.
0:57:54 So maybe after someone's listening
to this, maybe on your next weekend,
0:57:58 give it a little try, like build a app
you want it to build all along, maybe
0:58:03 React app, like throw in automerge,
throw in Yjs, see how far you'll get.
0:58:09 And I think the promise is
really that users, your end users
0:58:13 will get much better software.
0:58:15 And for you as an app developer,
your life becomes a lot simpler.
0:58:19 You can take off that, that big
brick and just enjoy drawing
0:58:23 and writing with your pencil.
0:58:24 That, that sounds super exciting.
0:58:26 And like having seen the ecosystem now
for the last two years evolve, I think we
0:58:31 have already come quite, quite a long way.
0:58:34 Now there is a real ecosystem
and you don't need to reinvent
0:58:37 everything by yourself.
0:58:39 If someone's like looking for some
references or some inspiration, can you
0:58:44 point to like any sort of software that
is already local first, or I guess like
0:58:50 local first is a bit of a spectrum.
0:58:52 Some apps might be more local first
than others, but is there like some
0:58:57 software that people already use today
that, that is directionally local first
0:59:04 that you can orient yourself around?
0:59:06 Just looking at the core set of apps on
your phone is like a nice starting point.
0:59:11 For web apps, there's things going
back all the way to like ether pad.
0:59:15 Right.
0:59:15 Where you have offline collaboration.
0:59:17 And then if you want to try
and build your own experience
0:59:21 automerge.org/docs/quickstart.
0:59:24 You know, you can get up and, you know,
experience that feeling of building
0:59:27 local-first software for yourself.
0:59:29 I think most apps that we from a user
perspective, see as working fast or
0:59:35 that you also trust to use them while
you're traveling on a plane, on a train.
0:59:40 I think they are all probably, even
though they don't have a local first
0:59:44 label yet on the software box, they
are probably already more local first.
0:59:49 And yeah, this is, I think,
whoever's trying to build something
0:59:54 next if you get to experience that
for yourself, I think that's the
0:59:57 easiest way to convince yourself.
0:59:59 I'm thinking of some software that
I think that I use on a daily basis,
1:00:04 whether like, whether it's like,
something like, Linear or something
1:00:08 like superhuman, those sort of apps,
while I'm not sure whether they were
1:00:12 from the inception, meant to be local
first, I think they've directionally
1:00:17 evolved in a way that is more and more
local first, there's probably still like
1:00:21 servers heavily involved, but in terms
of that client is getting a lot smarter.
1:00:27 I think that's already an
important step in that direction.
1:00:30 And that's certainly why
those apps feel great.
1:00:34 Amen.
1:00:35 Open Research Questions
1:00:36 So maybe before wrapping up, what are
some topics within Local First that are
1:00:41 very top of mind for you right now that
you spend a lot of time thinking about
1:00:46 and where you hope that we'll be a lot
further along in one or two years time?
1:00:51 Ooh.
1:00:52 I think for me, the biggest design
questions right now are around
1:00:57 versioning and version control.
1:01:00 One of the things we've found as we've
studied local-first software and talked
1:01:03 to writers and creators is that, um,
this idea of the cloud as like, Sort
1:01:10 of the panopticon where anyone can
watch you work is like really deeply
1:01:14 uncomfortable to a lot of people and
so exploring these ideas of creative
1:01:18 privacy and the ability to work the way
software developers work, which is you
1:01:22 work private until you're ready to share.
1:01:24 I'm really eager to explore
those ideas and bring those.
1:01:28 To the rest of the world, your
Google Docs, you should be able
1:01:30 to work in that same offline way.
1:01:33 So I think that whole...
1:01:34 Question about like bringing together
version control in new ways is a big one.
1:01:39 Authentication, authorization,
access control, sharing
1:01:44 permissions, that kind of stuff.
1:01:45 I think it's actually
quite closely related.
1:01:47 I think that's an area we're going
to see a lot of progress around.
1:01:50 I think the app models are going
to continue to evolve, right?
1:01:54 Like.
1:01:55 Auto merge is document based, electric SQL
is database based and Riffle is SQLite.
1:02:01 Both are actually right.
1:02:02 I don't think it's one or the other, but I
think exploring kind of that relationship
1:02:06 there is going to be a big area.
1:02:08 I think there's a lot of stuff
around security and encryption,
1:02:11 end to end encryption that's
going to unfold and with it.
1:02:14 What's that what's that quote?
1:02:16 I think it was Leah Kistner,
which is , cryptography is a
1:02:19 technique for turning many problems
into key management problems.
1:02:23 so I think we're going to see a lot
of that over the next few years.
1:02:26 I think that's just scraping the surface.
1:02:28 And I think all of this though
will also intersect with like
1:02:31 application development models.
1:02:33 Just thinking about other kind of
intersecting things, another big part of
1:02:37 where we're still relatively immature is
deciding what to sync and when to sync it,
1:02:43 There's this motto we have at Ink
and Switch, which is like, results on
1:02:46 the next frame are your money back.
1:02:48 We want everything you do to run at
a hundred hertz, none of this like a
1:02:52 hundred millisecond round time to the
server, and then you get to paint.
1:02:55 No, like when you interact with the
system, you should see the response
1:02:59 the next time your screen refreshes.
1:03:01 So that means on a hundred hertz screen,
you got 10 milliseconds to do everything.
1:03:04 And so if the server is 50
milliseconds away is too late,
1:03:08 you can't ask the server, right?
1:03:10 If you want to have things by
the next data, that means you
1:03:12 need to out there already get the
data before the user even asks
1:03:15 That has big implications on your
data architecture on like performance,
1:03:20 So many Yeah.
1:03:22 Things need to be in memory.
1:03:23 How do you manage memory use?
1:03:24 What about like low bandwidth connections?
1:03:27 So yeah, I think that whole area of
like sync optimization and background
1:03:31 sync and so on is really important.
1:03:33 And it's an area that have thought
about, but we haven't seen a ton of
1:03:37 progress on implementations yet, because
until you actually have enough users
1:03:41 and enough usage to really drive that,
1:03:44 Really interesting topics to dive
deeper on in future episodes as well.
1:03:48 There is one topic that also stands
out to me that I think we're not there
1:03:52 yet which is cross app collaboration
or cross app interoperability.
1:03:57 So as we're.
1:03:59 Steering in the direction where we
have like more and more AI in our
1:04:03 lives and all of those AIs so far are
looking also that they run on like
1:04:08 somewhere's remote server, et cetera.
1:04:10 And now we're going to feed more
and more of our data into those I
1:04:15 would love to see a world where like
the AIs can also similar, like the.
1:04:20 Like the programs that we're
building also run on our devices.
1:04:23 Now we need to make our
data available to those.
1:04:27 Have those programs and those AIs
collaborate between each other.
1:04:31 So in the same ways, like all
the stuff I have in my own head.
1:04:36 From conversations with you, from
conversations with others, my mind is
1:04:40 like able to, all of those different
interactions I have are almost like
1:04:45 little apps and they naturally can
collaborate yet my contacts app that runs
1:04:50 on my machine and my email app, unless
they're from Apple and does integrate it,
1:04:56 otherwise they can't like, what does that
integration story between apps, et cetera?
1:05:01 What does that look like?
1:05:02 And I think the foundation for that is.
1:05:05 The data needs to be on the device.
1:05:07 And now we need to build something on
top that makes it easier to interoperate.
1:05:13 That data is shared across apps.
1:05:15 That's a whole topic that I'm
also really excited about.
1:05:18 So we call that malleable software
at Income Switch, and it's a very
1:05:22 active area of research for us.
1:05:24 We've just published a paper called Embark
Today as of this recording about that
1:05:29 https://inkandswitch.com/embark where we
talk exactly about some of these problems.
1:05:35 One really cool thing about auto
merges design that I haven't seen
1:05:39 in other systems is that the grain
of the system means that any apps
1:05:46 which share a backup sync server.
1:05:49 can load each other's data as long
as they know how to interpret it.
1:05:53 And, you know, it's a little bit like
sharing a SQL database, but without
1:05:56 all the problems of sharing a SQL
database, because the database is not
1:05:59 like the shared resource that you have
to worry about people hammering, right?
1:06:02 you can just put data in a sync server
and then anything can query it out.
1:06:06 And so, I got to say, it's awesome.
1:06:09 I use it all the time.
1:06:10 When I break something, I can open
it up in another app and fix it.
1:06:13 You know, I have a whole suite of
little apps that are local first
1:06:16 that collaborate on our data now.
1:06:17 And yeah, well, we'll say more
about that sometime in the future
1:06:22 Outro
1:06:22 Peter, thank you so much for sharing
all of that about Local First with us.
1:06:28 I think this is the beginning of a
much, much longer journey where I want
1:06:32 to hear a lot more different voices,
like people building apps with Local
1:06:36 First, people building tools to help
other developers build Local First apps.
1:06:42 All the things we've just barely
touched the surface here today.
1:06:46 So thank you so much for coming on
the show and I'm looking forward
1:06:48 to having you back hopefully soon.
1:06:50 Yeah.
1:06:51 It was a lot of fun.
1:06:52 Take
1:06:52 Awesome
1:06:53 Thank you for listening to
the Local First FM podcast.
1:06:56 If you've enjoyed this episode and haven't
done so already, please subscribe and
1:07:00 leave a review wherever you're listening.
1:07:03 Please also consider telling your
friends about it, if you think they
1:07:06 could be interested in Local First.
1:07:08 Thank you again to Expo and Crab
Nebula for supporting this podcast.
1:07:12 See you next time.