Sponsored by
Show notes
Transcript
0:00:00 Intro
0:00:00 I feel like our bar as an
industry needs to be higher.
0:00:02 And I, I honestly think that web
development is to blame for some of that.
0:00:05 That's my spicy take because it makes
it easy for us to throw up our hands.
0:00:09 And say, I can't go past
that layer of abstraction.
0:00:13 Whereas in the native
world, you have a problem.
0:00:15 You can dig into those, the C binaries.
0:00:19 it might be hard, but you have the
power to go in and change things.
0:00:22 Whereas on the web, it's
like, it just works that way.
0:00:25 It sucks.
0:00:26 Like too bad.
0:00:26 I think we need aim, aim higher.
0:00:28 Welcome to the local-first FM podcast.
0:00:31 I'm your host, Johannes Schickling,
and I'm a web developer, a
0:00:34 startup founder, and love the
craft of software engineering.
0:00:37 For the past few years, I've been on a
journey to build a modern, high quality
0:00:41 music app using web technologies.
0:00:43 And in doing so, I've been falling down
the rabbit hole of local-first software.
0:00:48 This podcast is your invitation
to join me in that journey.
0:00:51 In this episode, I'm speaking to James
Long, the creator of the local-first app.
0:00:56 Actual Budget and the absurd-sql
project, which helped to pave the way
0:01:00 to bring back SQLite to the browser.
0:01:02 In this conversation, we go deep on
his journey, building Actual Budget,
0:01:06 including implementing a syncing
solution from scratch and expanding
0:01:10 from an Electron app to mobile and the
web while reusing most of the code.
0:01:15 Before getting started, also a
big thank you to Expo and Crab
0:01:18 Nebula for supporting this podcast.
0:01:20 And now my interview with James.
0:01:24 Hey, James.
0:01:25 So good to have you.
0:01:26 Hey, Thanks for having me.
0:01:28 So I've been a long time
fan of your prior work.
0:01:32 I think this has really been like
the first time where I've seen an
0:01:36 Actual local-first app pun intended.
0:01:39 You've been working on the Actual
Budget app in the past, and which
0:01:43 has led to quite a few, technical
innovations, particularly for the web.
0:01:48 So that's what I'm looking forward to
exploring today, but I'm curious what has
0:01:53 led you to exploring local-first and what
has led you to work on Actual Budget.
0:01:58 Electron/Actual
0:01:58 Sure.
0:01:58 So first of all, I'll just state
that Actual Budget is, uh, I
0:02:01 don't actually work on it anymore.
0:02:03 And I open sourced it about two years ago.
0:02:05 And so the community has taken it
over and done a great job with it too.
0:02:08 Um, but I started it around 2017.
0:02:11 And back then I just really wanted
a local app, just the web back
0:02:16 then was even worse than it is
now, just like development-wise.
0:02:19 And I don't know, I, it
just, It didn't excite me.
0:02:22 It's a personal finance manager.
0:02:23 It felt like it should be local.
0:02:24 It felt like I should be able to throw raw
SQL queries at it and get my data back.
0:02:30 So it just felt like a very good fit.
0:02:32 And using Electron back then was
amazing because I could just like
0:02:35 load up SQLite and load a SQLite
database from a local disk and use
0:02:41 the native SQLite C bindings, right?
0:02:43 It was, it was fantastic.
0:02:44 So I built a basic local app just
because I wanted it to be local
0:02:48 and I just didn't want to deal
with like hosting it somewhere and
0:02:50 I was like, this is just for me.
0:02:51 This is just a fun thing
for me for certain apps.
0:02:53 I've always liked the idea of it.
0:02:55 Just like being super local.
0:02:56 Uh, you, you own everything.
0:02:58 It's not dependent on anything else and
that you just have raw access to the data.
0:03:02 Obviously, at some point, I hit this
problem where it's like, well, shoot, if
0:03:05 I drop my laptop, All my data is gone.
0:03:07 Right.
0:03:07 or like my wife just wants to check
one charge somewhere and she's on
0:03:11 her laptop and she can't do that.
0:03:13 And so, uh, eventually I was like,
well, crap, if I want to make a
0:03:16 business out of this, especially I got
to solve the collaboration problem.
0:03:20 And so that's what led
me down into syncing.
0:03:22 And so I investigated a couple of things
and it worked out really, really well.
0:03:24 And so I went down that path from
there and I just continually.
0:03:27 Invested into it.
0:03:28 And it was fun.
0:03:29 That is super impressive, particularly
given that you've already started
0:03:32 working on that in 2017, just for
reference, the official local-first essay
0:03:38 by Inc&Switch, uh, came out in 2019.
0:03:41 So it looks like you've been on the same
journey, uh, maybe aware, maybe unaware
0:03:46 that other people have been also exploring
the ideas of local-first, and then you've
0:03:51 just arrived on very similar conclusions.
0:03:54 So I'm curious to learn more about
the technical challenges that
0:03:59 you've been facing, really, uh,
innovating on so many fronts there
0:04:03 to make this, this vision a reality.
0:04:06 And I would like to better understand
also one aspect you've been mentioning
0:04:10 that the, uh, App should work local.
0:04:12 Are you referring to an app here as a
desktop app from something like Electron?
0:04:17 Or are you also talking about a
more of like a progressive web
0:04:21 app that runs also in the browser?
0:04:23 So back then at the time, I was mostly
thinking just like a desktop app.
0:04:27 I want to see the icon in the doc.
0:04:28 I want, I wanted it to just be native.
0:04:31 Native-ish.
0:04:32 I know we're kind of faking that
with Electron, but I wanted just
0:04:35 an app that I can click and I
open my finances and I search and
0:04:38 then I do command Q and it's gone.
0:04:40 So back then it was more a desktop app.
0:04:42 And eventually I did port
everything to the web.
0:04:44 And that was a whole nother thing because
it's really hard to compete with the web.
0:04:47 I mean, just not only
distribution, uh, but just like,
0:04:51 that's just where everybody is.
0:04:52 So it's evolved a little bit more
to now mean probably Like when I
0:04:56 say local app, it could totally be
a web app as well, or a mobile app.
0:04:59 Like all of it should just be
something that can work locally.
0:05:01 I think the web still needs to catch
up in a lot of ways to get there.
0:05:05 But at this point, I, when
I say it, I mean like a.
0:05:09 SQLite as Data Layer
0:05:09 So before diving a little bit more into
the nitty gritties of the technological
0:05:14 choices that you've made, you've
already mentioned that you've been
0:05:17 choosing SQLite for the data layer.
0:05:20 So I think the, the web world is.
0:05:23 Sometimes like divided across
front-end and back-end, I think using
0:05:28 a SQL database is much more common
on the, on the back-end and on the
0:05:33 front-end, I think you're more used
to things like Redux, MobX, et cetera.
0:05:39 And given your use case of
an Electron app, this is sort
0:05:43 of like a murky in between.
0:05:45 So I'm very curious to hear
more about your intuition.
0:05:49 What led you to wanting to use
a SQLite database that I guess
0:05:53 in the realm of Electron rather
falls into the front-end realm.
0:05:59 So I'm curious what led
that design decision.
0:06:01 Well, like I said, I mean, for the data
layer itself, it was just so obvious
0:06:05 to me that it's a small set of data.
0:06:07 Some, some people worry about SQLite not
scaling well to like millions of rows.
0:06:11 I think I have at this point, nine years
of my transactions in there and I, I'd
0:06:15 have to count, but it's, I think it's
in the like tens of thousands, uh, aims.
0:06:18 So this is like not a large set of data.
0:06:20 SQLite would just seem like
such the perfect fit for it.
0:06:22 But then once you have SQLite and it's
a local app and then you're writing in
0:06:27 front of components, it seems silly to
have to design this API layer that's
0:06:32 either like a WebSockets messaging layer
or like an HTTP like URL based API.
0:06:38 Like it seems purely you're going
through the motions to have an API
0:06:43 that literally just is intercepted
locally and then runs the data locally.
0:06:47 So once and then like you get that.
0:06:49 Data back as like JSON.
0:06:50 So you, you query something with
like an HTTP or like web socket
0:06:53 API, and then you get the data back.
0:06:55 It's like you, why not just go ahead
and just like write SQL queries, right?
0:06:59 In those components.
0:07:01 It literally makes no difference
because it's intercepted locally.
0:07:04 So then I started kind of,
Exposing SQLite even more and more.
0:07:07 And I ended up coming up with my own
little data, data querying language,
0:07:11 because when you're doing really
quick stuff, it is nice to, it's not
0:07:15 an ORM at all, but it's basically
like a query builder type thing.
0:07:19 It's, it's a pretty common thing
that I think a lot of people do.
0:07:21 I think there's a library called Knex.
0:07:23 K N E X that like helps you
build sort of like that.
0:07:26 But I, I built it because I like
building stuff myself and it was, it
0:07:30 was a pretty interesting thing, but it
does basically let you construct SQL
0:07:32 things and then it hands that to the
backend and it knows how to execute it.
0:07:35 The other thing that it knows how
to do is it knows how to do like
0:07:38 live queries, which you don't get
with like raw SQL queries, unless
0:07:40 you parse the query or something,
but it knows which tables to watch.
0:07:44 And so as other syncing messages come in,
the data will just automatically update.
0:07:47 And so, yeah, it just felt
like a really nice fit.
0:07:51 Since everything is local anyway.
0:07:52 That sounds super compelling.
0:07:54 And I think we've arrived at a few
similar conclusions given that with my
0:07:59 work on Riffle and LiveStore, I've also
built some similar aspects such as like
0:08:04 the reactivity that you've pointed at.
0:08:06 I'm curious, like how you actually
went about implementing that since
0:08:10 for the listeners who might not
be as aware of all the SQLite
0:08:15 internals, SQLite itself doesn't
really help you much with reactivity.
0:08:20 There is a few hook points, but,
uh, you get to roll quite a bit
0:08:24 of the stuff yourself on top.
0:08:26 So I'm curious how you went about that.
0:08:28 Yeah, that part.
0:08:29 Honestly, it's not super innovative.
0:08:30 I actually remember around 2017
when I started doing this kind
0:08:34 of a stuff, I went really deep on
SQLite internals and there's a hook.
0:08:38 I forget what it's called, but
it's like SQLite underscore pre
0:08:42 post update or something like that.
0:08:44 There's a hook that you get that gets
called whenever, whenever an update
0:08:47 happens and you get like the data
before and the data after, but it turned
0:08:51 out to just be a really weird hook.
0:08:53 When you want to do live
reactive stuff, you, you want
0:08:55 it to be somewhat fine grained.
0:08:57 It doesn't have to be super fine
grained, but if, if you want A change
0:08:59 happens, then the, the entire, like
all of the data on the app re renders.
0:09:03 SQL is fast and it's
local, but it's still slow.
0:09:06 If you're going to re render every
single time you change one little thing.
0:09:10 And so you want to make sure that you'd
somewhat scoped to like the table that
0:09:13 changed or something like that, at least.
0:09:15 And that hook, like it
provided back like weird IDs.
0:09:19 Like I could never figure out how
to actually get the data itself
0:09:22 that changed, like this column.
0:09:24 And this row changed from X
to Y it was like, I couldn't
0:09:28 even get like that basic stuff.
0:09:30 And maybe I was just doing it wrong.
0:09:31 There are some hooks in
there that seem promising.
0:09:34 But I think from what other people are
doing, they're using like, like the wall
0:09:37 file to do like really interesting things.
0:09:39 I think that seems like a
more reasonable approach.
0:09:41 But overall, actually, honestly,
it's pretty, it's pretty basic.
0:09:45 All updates have to go through an API.
0:09:47 And so unfortunately, if you open up the
SQLite file locally, you can update it.
0:09:52 But those messages will
not be synced around.
0:09:55 You'll have to like reset the file
across all devices, which it's possible.
0:09:59 And people have done that before, if they
really want to like mess with something.
0:10:02 But, um, it's not a thing where
you can open up the SQL file and
0:10:05 update things directly from there.
0:10:06 You have to call the update function,
which then creates a bunch of like CRDT
0:10:11 stuff, sends it out, and then actually.
0:10:13 I take that back a little bit.
0:10:15 You need to call the update function to
generate the CRDT, like the sync messages.
0:10:19 And they go out to the server
and they get applied locally.
0:10:22 It's the part of the pipeline
that applies the messages,
0:10:24 which does, does the reactivity.
0:10:26 That allows me to kind of
watch for changes, right?
0:10:28 Because All mutations through the system,
even if they're just local mutations, go
0:10:33 through this like CRDT messages system.
0:10:36 And so you can see when column X
on table transactions changes, it's
0:10:41 changing from this value and I have
the before value and after value.
0:10:45 And so when those messages get pumped
through is when it fires off all
0:10:49 of the listeners that are listening
for that small piece of data.
0:10:53 And yeah, that's basically how it works.
0:10:55 Got it.
0:10:55 So you've captured most of that
and probably in, in JavaScript.
0:10:59 And since you're all using, you're
building the library, you're building the
0:11:03 app, you know how to use it correctly.
0:11:06 So you've built a lightweight
query builder on top.
0:11:10 How are you using that query builder then?
0:11:12 Um, have you like wrapped
those in React hooks?
0:11:15 Or how are you wiring up your data
into the UI directly at some point?
0:11:20 Yeah, there's a useLiveQuery() hook,
and then there's a usePageQuery() hook.
0:11:25 The page query one is interesting
because it allows you to like page
0:11:28 in results, and it returns an object
that has like a dot next function.
0:11:32 I mean, it just knows how to automatically
add the limit and, and, and offset.
0:11:35 And it does some other
pretty fancy things too.
0:11:37 And so they're just hooks
that Knows how to rerender the
0:11:40 component when the data changes.
0:11:43 Got it.
0:11:44 I mean, uh, I've gone through a similar
journey of where I used like some
0:11:49 state management libraries and react
front-ends in the past, and then,
0:11:53 uh, actually being able to use and
embrace SQLite for all its benefits and
0:11:57 the front-end is quite magical, like
little tricks that I found to be super
0:12:01 compelling is that I can actually like.
0:12:03 Touch the SQLite file, whether I can
like look at it and see the values being
0:12:08 updated by the app, or also like go to
it and like delete it, reload the app.
0:12:13 And I'm starting from scratch.
0:12:14 Those little things, they're
just super compelling and make
0:12:17 it super fun to work on the app.
0:12:19 Is there some things like that, that
you found that just gave you like a
0:12:24 really nice boost in your development,
like velocity and productivity and fun?
0:12:29 Best Practies
0:12:29 Sure.
0:12:29 Yeah.
0:12:29 I mean, it's.
0:12:30 It's totally true that it kind of
forces it to be like, if you're not
0:12:33 using Postgres and you're using SQLite
instead, and they're all files, it's
0:12:37 so easy to just like move things
around and create like a fresh file.
0:12:40 The, the demo app literally
just copies like a demo.
0:12:42 SQLite file into the
app's like database file.
0:12:46 And if you're using Postgres, that
kind of stuff just gets a lot harder
0:12:49 because it's so like process oriented
and like process based, you can't
0:12:52 just easily do that kind of stuff.
0:12:53 But like, and yeah, being, being
able to dig into things and do
0:12:56 SQLite queries, it felt really fast.
0:12:59 I think you, You know, if you're
developing the app, you have
0:13:01 access to the Postgres database.
0:13:03 So you can do that kind of stuff as well.
0:13:04 But one of the things that was, does
come to mind is when I was building
0:13:08 the Electron app, I had this strategy.
0:13:10 And I think I wrote a post
about this, where there is a,
0:13:12 there's actually two windows.
0:13:14 One window is for the front-end and
one window is actually for the backend.
0:13:17 So the backend does not run
in a, like just a node process
0:13:21 that is, you know, Invisible.
0:13:23 It actually runs in another Electron
window with like node integration set
0:13:27 to true, and it can access node APIs.
0:13:30 What's cool about that is
that I can open the dev tools.
0:13:32 I expose a bunch of stuff
to the top level, and then I
0:13:35 can like query around stuff.
0:13:37 I can query the database directly.
0:13:38 I can get objects back and
get it that really nice Chrome
0:13:41 dev tools, like object viewer.
0:13:42 Um, I can track performance.
0:13:44 So I can click the performance
tab and click start.
0:13:46 Start performance recording,
do a bunch of stuff on the UI.
0:13:49 And then like, I can set,
um, stop the performance.
0:13:51 You can do performance tracing.
0:13:52 So you can actually start tracing
how long the SQL queries took.
0:13:55 So being able to access the backend dev
tools or the Chrome dev tools for the
0:13:59 backend, and like being able to interact,
like directly interact with, with the
0:14:03 database was like super, super great.
0:14:05 Yeah, that, that sounds amazing.
0:14:07 So you've been using SQLite with Electron,
but you've already hinted at before
0:14:12 that at some point you realize, okay, a
single Electron app is not quite enough.
0:14:18 We also carry phones.
0:14:19 We might carry other devices.
0:14:21 Um, What if your, your
single device gets lost?
0:14:24 What about all of your app state?
0:14:26 So I'm curious at which point
you've then found your way to, to
0:14:31 also implementing collaboration or
syncing and how you went about that.
0:14:35 So I, I can't remember when it
was exactly, maybe 2018, 2019, I
0:14:40 started looking into this and I.
0:14:43 Went about it by just hearing things
that my friends were talking about
0:14:47 that were kind of interesting.
0:14:48 And so they were, they were doing like
Raft and some of those protocols in
0:14:52 the backend and kind of just, they
were deep into that kind of area.
0:14:55 And so I heard, honestly,
I almost gave up.
0:14:58 I was like, this just, all of this
stuff seems way, way too complicated.
0:15:01 I do not have time to do this.
0:15:02 But then I ended up just poking
around to see if there was any
0:15:05 possibilities of things I was doing.
0:15:07 I had this like a really initial
implementation, which was super naive.
0:15:10 I, there's a gist that I explained
it somewhere and it was like
0:15:13 really, really easy to pick up.
0:15:15 Pick it apart.
0:15:15 Once I started implementing it and
like running a test against it, like
0:15:18 there was like, I think it was sort of
operational transform based, but it just
0:15:22 like was, it fell apart way too easily.
0:15:24 And I was like, this is really hard stuff.
0:15:25 And when I looked at CRDTs, like I
could never grasp what they were.
0:15:30 Nobody really, I, back then, at least
it was just not a good, it was too much
0:15:33 math, too much, very, very intimidating.
0:15:36 And they're really not that hard.
0:15:37 Like it does not need to be explained
that way, but something clicked at
0:15:40 some point when I finally implemented.
0:15:42 A basic last right winds map that
the thing that really unlocked.
0:15:46 It was hybrid logical clocks.
0:15:47 I can't remember where, like, when
I found that, but when I started
0:15:51 looking into that and reading that
paper, uh, there's a simplicity about
0:15:54 them that I found really compelling.
0:15:56 And it really matched match my
own technical kind of approach
0:15:59 for things, which is like.
0:16:01 make things as simple as possible and,
and, and the least surprising as possible.
0:16:05 And you get a lot of benefits from that.
0:16:06 Now there are, there are drawbacks
to HLCs, but the benefit of them,
0:16:11 especially for this use case
seemed like a really good match.
0:16:14 And so I went off and, you know,
it was one of those things where
0:16:17 like everything kind of came
together in a couple of weeks.
0:16:19 And I started seeing some really
compelling success with that.
0:16:22 And also the ability to unlock
things like undo and Yeah.
0:16:25 Redo, because once you start using the
system that like mutates everything
0:16:29 through these like messages, you can
suddenly start tracking those messages
0:16:33 and you can invert the messages.
0:16:35 So undo literally becomes take this
batch of messages that happened
0:16:39 in the last action and invert them
and then apply those messages.
0:16:43 And suddenly, Actual turned out to
have a really robust undo and redo
0:16:47 system, which I'm super proud of.
0:16:49 And like, work, like literally
everything that you do in Actual,
0:16:51 you can press command Z to undo.
0:16:53 If you import 2000 transactions and
it runs a bunch of rules and mutates
0:16:57 them, press command Z and in like 500
milliseconds, or not 500 100 milliseconds.
0:17:03 Um, everything will go
back to where it was.
0:17:05 So I started seeing signs of
this architecture, which was
0:17:10 like really exciting, and then
I just kind of went from there.
0:17:12 So just as a side questions for, for those
of us in the audience who might not be
0:17:18 familiar with hypological clocks, could
you give a quick explainer, uh, what the
0:17:24 Hypological Clocks
0:17:24 Sure.
0:17:25 So I'll try to be fast.
0:17:26 I think we could probably talk
about this, uh, area of research
0:17:30 probably for the rest of the time.
0:17:32 Hybrid logical clocks is a way to
solve the coordination problem.
0:17:35 So the problem with distributed systems,
which a local-first app is a distributed
0:17:39 system, because you have copies of the
app and copies of the data across multiple
0:17:44 devices, is you need to know when, if you
have something like a last write wins set.
0:17:48 If two people write that
offline, then come back online
0:17:50 to sync up, which one wins?
0:17:52 So you, you, you have to have a clock
for every single mutation in the system.
0:17:56 There's a vector clocks.
0:17:57 There's a more advanced clocks that are
built on top of things like vector clocks.
0:18:00 There's a lot of different approaches for,
to, to solve this kind of a problem, to
0:18:03 say, which one came after the other one.
0:18:06 To take a pretty simplistic approach
here, but it's still super robust.
0:18:09 And the very short summary of it is
it actually, the neat thing about it
0:18:14 is that it serializes into a string.
0:18:15 Um, And so the comparison of if this
message came after or before is you just
0:18:19 compare the string, like it says less than
or greater than the string is always the
0:18:24 same length and so it's lexically ordered.
0:18:27 So you can just say, if I wanted to
inspect my database and get the messages
0:18:31 in order, I would say, select star for
messages and then like order by the.
0:18:35 CRDT, HLC, and then everything
comes back ordered, right?
0:18:39 Whereas like vector clocks and all these
other complicated ones are like complex
0:18:42 object data structures that have to keep
track of the ID or the counts or whatever
0:18:46 of every single device in the whole world.
0:18:48 And you have to like name, name those
devices and, and, and do a lot of things.
0:18:52 HLCs are a thing that Take the
current time, like the clock of the
0:18:57 system, which sounds terribly scary.
0:18:59 If you know anything about this
thing, like involving the clock
0:19:01 of the current local computer, it
sounds awful, but they do it in a
0:19:04 way that is really, really novel.
0:19:06 So the first part of the string
is like the, the UTC, like that
0:19:10 timestamp in the format, that's
like that Z at the end, right?
0:19:13 That like, I forget what it is.
0:19:14 And so it's that, it's that big
timestamp and then it's dash.
0:19:18 And then there's like
a, a padding of bits.
0:19:21 And then there's another like dash.
0:19:23 And then there's the device ID,
I think, at, at, at the very end.
0:19:27 And so the way that you get
deterministic order is that,
0:19:31 that little bucket of bits in the
middle, that's the key there, right?
0:19:35 So that bucket of bits
represents an integer.
0:19:38 When you receive a message,
normally you use your local time.
0:19:41 Well, that's not exactly true.
0:19:43 You actually use your local time.
0:19:45 Or the last highest time that you've
ever seen from the whole system.
0:19:50 So if you are receiving a bunch
of messages, you're reading the
0:19:53 times off of those messages.
0:19:54 And you say, if that time is greater
than my time, so that laptop's
0:19:58 clock is like one minute faster,
then I'm going to have a timestamp.
0:20:02 One minute faster.
0:20:02 I need to, I need to fast forward
my time to match that clock's time.
0:20:06 Cause that was like that person's
clock is later than mine.
0:20:09 But when you start doing that kind of
a stuff, you can't generate another
0:20:12 message with the same exact time, right?
0:20:14 You need to differentiate those
two messages locally somehow.
0:20:16 And so you increment that.
0:20:18 That number that's stored in
those bits in the middle by one.
0:20:23 And so if things are generating with
the same timestamp, well, suddenly that
0:20:26 timestamp is not ordering things anymore.
0:20:28 Now it's the little set of bits in
the middle that are ordering things.
0:20:31 And you can serialize them as like
a hex value, I think, or something
0:20:34 like something that's still like a
string that can be lexically ordered.
0:20:37 And so you're bumping that up, but
the minute that your, your system
0:20:40 moves forward, like a second.
0:20:42 Or, or some, some amount of time, you
can reset that back to zero because
0:20:46 the, the, the minute that your system
meets up with the time that you had
0:20:50 seen from the other system, and then
you can start using your local system
0:20:53 again, then you can start using, then
you reset the counter to zero and you
0:20:57 don't need to use that anymore, right?
0:20:58 So it's, it's really tricky to, this
is really interesting technique where.
0:21:02 You can sort of leverage your
local time and other everybody in
0:21:06 the system can coordinate on this
and you get this really simplistic
0:21:09 approach throughout this whole thing.
0:21:11 Now, the big downside of this is that
you can't have something that's so far
0:21:15 ahead in time that you, your timestamp
locally is just meaningless now and that
0:21:19 you're always incrementing the bits.
0:21:21 You're going to hit a ceiling where
you can't, that integer is too large.
0:21:24 And you can't store it in like the,
you know, whatever the 24 bits it is.
0:21:28 So then you're screwed.
0:21:29 Then that thing is all busted.
0:21:31 So there's this whole like
thing where there's approach.
0:21:33 You can say like all of the
devices in this system need to be
0:21:37 synchronized with at least like
five minutes or like an hour.
0:21:41 And if you try to generate or
send a message or get a message.
0:21:45 That is outside of that
timeframe, you just reject it.
0:21:48 So it's a little bit of a like
brute force simplistic approach.
0:21:53 Uh, but for something like Actual,
it worked out really, really well.
0:21:56 It would not work in a complex
distributed system where there's
0:21:59 like many, many, many clients.
0:22:01 And you know, they're, they're,
it's not as robust for sure.
0:22:05 That's as simple as I can explain it,
hopefully as, as a little bit longer
0:22:08 than I was hoping, but that's, that's,
that's the best way that I can help.
0:22:11 No, this was super insightful.
0:22:13 And I think going deep into those
kinds of topics, I think that's, that's
0:22:17 what the audience is interested in.
0:22:19 So thanks so much for, for
like taking that little detour.
0:22:23 I understand that now much better.
0:22:25 So you've taken this concept and then
went from your local SQLite database
0:22:31 And how did you take Hyperlogical
Clocks with your SQLite database
0:22:36 and now made things collaborative?
0:22:38 So once you have Hyperlogical Clocks,
it becomes pretty easy to sync things
0:22:42 around because I essentially, honestly,
I don't use a super complicated.
0:22:47 I know, is it Martin Kleppmann?
0:22:48 I think works on like a lot
of really, really robust data
0:22:52 structures that work really well.
0:22:53 When you're in this distributed
world, you have to make sure that
0:22:56 things don't end up in a bad state.
0:22:57 So if you want like a.
0:22:59 Tree data structure.
0:23:00 And you want to say, Hey, this, like,
there should never be any orphaned
0:23:03 nodes in this tree tree data structure.
0:23:05 Well, it's really easy to get
that state in a naive thing.
0:23:08 If you say like a last right, when set
that it's like parent child, and then
0:23:12 like one person updates a node and the
other person had deleted that node.
0:23:17 And so the node gets like.
0:23:19 removed from the parents, but
then the message comes in later
0:23:22 that this person edited that node.
0:23:24 So the node gets created again,
but it's an orphaned node.
0:23:27 Like that's a weird place to be.
0:23:29 So like there's work in the CRDT
world, which is fantastic and makes
0:23:33 those kinds of things super robust.
0:23:35 I did not use any of that kind of stuff.
0:23:36 So like those kinds of things, I just kind
of accepted and was like, well, I just
0:23:39 code defensively against like bad data.
0:23:41 And generally speaking in my app,
I didn't have a ton of places where
0:23:45 things needed to be super robust.
0:23:47 It was pretty easy to defend against
bad data, but essentially like I said,
0:23:51 there's like an update, a crate and a.
0:23:53 Delete function, they all actually
intercept to a single like lower level
0:23:58 update function because instance and
updates are exactly the same thing that
0:24:02 take a, take an object of things to set.
0:24:04 And that object has to contain an ID.
0:24:06 It turns it into CRDT messages.
0:24:09 So every single field set.
0:24:10 So if you said, set the transaction
amount in the transaction date to
0:24:13 X and Y, those would become two
different messages with the same.
0:24:18 object ID, right?
0:24:20 Like they're, they're
targeting the same object.
0:24:22 One is set the field amount to X and
one is set, set the field date to Y.
0:24:26 There's two messages get created and then
this gets sent off to a syncing server.
0:24:30 The syncing server is
really, really stupid.
0:24:32 It just holds the messages and
when it's, when a client asks for
0:24:36 messages, it gives us messages back.
0:24:38 And so it can ask for messages
since a certain point in time,
0:24:41 which again, in our, uh, HLC clock
world, that is our point in time.
0:24:46 So it's literally just a Postgres query
that says select star for messages
0:24:50 where HLC is greater than X and X
is the HLC that you gave it, right?
0:24:55 So, so nice that you can just do
that, like greater than comparison.
0:24:58 And then it, so it gets the messages back
and then it says, Hey, I'm, I'm, I'm,
0:25:01 I'm all synced up and that's essentially.
0:25:03 How it does.
0:25:04 There is like a Merkle tree here in there.
0:25:08 Merkle Trees
0:25:08 Yeah.
0:25:08 Let's get into it.
0:25:09 Uh, how Merkle tree is fitting in here.
0:25:11 Sure.
0:25:12 So yeah, the, so there's
a big problem here, right?
0:25:15 Like, how do you know that
you're actually in sync?
0:25:18 How do I know what messages to ask for?
0:25:19 So when I, when I open up Actual and I
hit sync, do I ask for all the messages
0:25:24 in the entire history of the world?
0:25:26 Or like I could ask for.
0:25:28 The messages since I've last opened
the app, like that seems nice, or like
0:25:32 the last time I have synced, right?
0:25:34 And then I would get those
messages and apply them.
0:25:37 But there's like several questions there.
0:25:39 One, what if another client had created
a message and just hadn't synced it
0:25:43 yet, and created the message before
you last synced, and then you closed
0:25:47 the app and like didn't open it.
0:25:48 And then the other app, Came up, came
up and synced and then it like sent that
0:25:52 message into the system and then you
open, you know, the, the, the original
0:25:56 app, you would miss that message because
I've, I've seen since, since I've
0:25:59 last synced, there couldn't possibly
any, couldn't possibly be any more
0:26:03 messages, but that's not true at all.
0:26:05 In this distributed world, you have
to assume everything bad, everything
0:26:08 out of order is going to happen.
0:26:10 You cannot code like that.
0:26:12 The other problem.
0:26:14 Is when I have synced up, like, let's say
I asked for all, all of the messages in
0:26:18 the world and I apply them locally, how
do I know that I'm just actually valid?
0:26:24 Like there could be a bug in my system.
0:26:26 And so like, how do I know that
I have, like, when you, when you
0:26:29 get all of those messages back,
how do I know which ones to apply?
0:26:32 So locally, there's like kind of a
ledger of things that you've applied.
0:26:35 And so you, you, you go through
every single message and you
0:26:38 say, have I applied this message?
0:26:39 I've already applied it.
0:26:40 Don't apply it.
0:26:41 If I have not applied it and it is a valid
message, like if it's a last right, when
0:26:46 set, it'll say, if this has been written
by a message, like later than this one,
0:26:52 then I can just describe this as well.
0:26:54 So there's logic about how
to apply the messages there.
0:26:57 That could be buggy or just like
a network request is so weird.
0:27:01 And there's a state that I
just, I didn't anticipate.
0:27:03 You need look kind of like a, a, another.
0:27:06 piece of data structure, and
that's, that's, that's the Merkle
0:27:08 tree to sort of audit things.
0:27:11 It's, it's, it's a hash of
everything in a system, right?
0:27:13 So if you hash all of the objects, all
of the CRDT messages, you can think of
0:27:18 all the CRDT messages as like leaf nodes.
0:27:21 And then you have like them grouped into
buckets and every node in the tree, all
0:27:26 the way up to the root is a new hash.
0:27:28 And so the hash at the root is a hash
of everything in the entire system.
0:27:31 So you can quickly compare if like,
have I seen all of these CRDT messages?
0:27:35 I just compared the two root hashes.
0:27:37 And then if I, if those hashes are
exactly the same, then I know I've
0:27:40 like, these have been processed.
0:27:42 Where it gets weird is that like,
I use, A base three system, I think
0:27:48 of like, basically every node in
the tree is a zero one or two.
0:27:52 And basically you can construct
a timestamp of basically
0:27:57 how I bucket the messages.
0:27:59 Like I talked about buckets, right?
0:28:01 You have to have like,
what are those buckets?
0:28:03 The buckets for me were
basically time windows.
0:28:06 And so I had time windows of
like, down to like, 10 minutes.
0:28:10 So every single like CRDT messages
all applied within a 10 minute window
0:28:15 would be hashed together into one hash.
0:28:17 And that would be one
leaf node in the tree.
0:28:20 And those would be all
the separate leaf buckets.
0:28:22 So the bucket above it represented
a new, a larger window.
0:28:26 Of time, right?
0:28:27 And it was a, because it was base three,
you could reconstruct this timestamp.
0:28:31 The thing that I wanted to guard
against was these are a lot of messages.
0:28:34 There could be tens of
thousands of messages.
0:28:37 You have to come up with a, a system
that is detailed enough so that you're
0:28:41 not requesting too much, too, too, like
too many messages, but it needs to be.
0:28:46 Coarse enough so that the tree
just doesn't get massive because
0:28:49 this tree is sent across the
network every single time you sync.
0:28:52 It's stored in your database and
updated, like updated as a blob
0:28:56 every single time that you sync.
0:28:57 So it cannot be a huge tree.
0:28:59 And so this base three system encoded
these windows in a way that allowed
0:29:03 me to calculate things down to the
five minutes windows and not be huge.
0:29:07 Like I think it could ever
only get like 10 leaves deep.
0:29:10 And so that solves the depth problem.
0:29:12 It doesn't solve the.
0:29:13 breadth problem of the tree.
0:29:14 So the Merkle tree could still
get really wide if you're using
0:29:17 the app over years and years.
0:29:18 So if you think of this huge tree, we're
ever seeing in windows of time, there's
0:29:21 only ever two paths down 10 nodes deep.
0:29:25 And before that, it just
prunes them all away.
0:29:28 It just deletes them.
0:29:29 What that means is that if I miss a
message, like I think it was about a
0:29:33 nine month mark, that if a message comes
through past nine months previous to that,
0:29:38 I don't know which things to download.
0:29:41 I check that I've received all of the
messages by comparing the hashes, right?
0:29:45 If the hash of the root is wrong,
then I go down and I compare.
0:29:48 That's how I get the window
to retrieve messages by.
0:29:50 So I compare down and I get, I get
the, the node that is different.
0:29:54 And then from that different node,
I can construct a window and says
0:29:57 something came through like, like
a message from like, like three
0:30:01 weeks ago, there's something here.
0:30:02 That's a difference.
0:30:03 I need to, I need to retrieve all of
the messages from three weeks ago, and
0:30:06 I need to reapply them all because like
something changed their Merkle tree hat.
0:30:10 Like the, the server sends the
Merkle tree, it's Merkle tree to me.
0:30:14 And so I can compare the service Merkle
tree to mine and say, Hey, the service
0:30:18 Merkle tree like changed around this time.
0:30:20 So I need to get those messages.
0:30:22 There's a certain point in time, which if
a message, if the Merkle tree like gets.
0:30:25 Change like a long time ago, like
a really old message comes to the
0:30:29 system and just screws everything up.
0:30:31 I actually won't know that because
I've already pruned that tree away.
0:30:34 So you'll either have to redownload
all of the messages, which is
0:30:37 probably not practical, or you just
reject that client and you like
0:30:40 detach it from the whole system.
0:30:42 So there's edge cases there that
you have to sort of think about,
0:30:44 but the really nice thing that the
Merkle tree gives me is that window.
0:30:48 And a validation that I have actually
processed, like if the roots hashes
0:30:52 are the same on the server and the
client, then I know I'm up to date.
0:30:55 So thank you so much for giving
this, uh, quite in depth overview
0:31:00 of how Merkle tree works and more
specifically how you applied them
0:31:05 on the, the Actual syncing scenario.
0:31:07 That was very insightful.
0:31:09 I've now seen Merkle trees applied on
a, on a few different technological
0:31:13 scenarios and the way how you've now
used them with the time buckets, I
0:31:19 think is super elegant where it can,
in the happy path, save a lot of
0:31:23 work, uh, for, for the syncing engine.
0:31:25 And once you've understood the entire
system, actually quite simple to think
0:31:29 about, and I'm sure also when you, maybe
something went wrong at some point and
0:31:33 you at least still had sort of like an
intuitive mental model, how you can debug
0:31:38 this, et cetera, and how you can test it.
0:31:40 So I'm sure you didn't.
0:31:41 just build a syncing system just for one
or two Electron apps, but probably for
0:31:48 a vision to go beyond the Electron app.
0:31:54 Beyond Electron
0:31:54 from here?
0:31:54 Yeah.
0:31:55 So once I validated that I was going
to build a syncing engine and that
0:32:00 it worked, I built a mobile app.
0:32:02 Because obviously that was part of the
reason for wanting to do syncing is to
0:32:06 be able to support like a mobile app.
0:32:07 And I sort of naively was like, I'm
not going to build a shoddy mobile app.
0:32:11 I'm going to, I'm going to do
this and I'm going to do it right.
0:32:13 And so I, I, I did use React Native.
0:32:15 I didn't, I wasn't so naive that I was
going to build a separate Android and iOS
0:32:20 at myself, but I did attempt to use React
Native and I built it with React Native.
0:32:24 And then I leveraged a project that
was at the time was having a decent
0:32:27 amount of maintenance and investment
from, I forget which company started
0:32:31 it, but it's a project called Node.
0:32:33 js mobile.
0:32:35 And so they basically took Node and
they compiled it for iOS and Android.
0:32:38 And they basically built this thing
that would load your JavaScript bundle.
0:32:42 And you could write JavaScript
based off of Node APIs.
0:32:45 And it was great.
0:32:46 Cause like HTTP, like all of the
standard node APIs that you would
0:32:50 expect worked there and it fired off
as like a separate process, like it
0:32:53 did it in, in the, in the proper way.
0:32:55 And so I got a prototype working
and it worked really well.
0:32:57 And so I was able to use
the exact same backend.
0:33:00 And then I built a new
front-end in react native.
0:33:02 That was like mobile specific.
0:33:03 Cause I was like, I really want to
think this through in like a mobile way.
0:33:06 And I'm not going to like shrink
down this transactions table and from
0:33:10 desktop and like, Try to shove in
this complicated table on the mobile.
0:33:13 I'm going to have a proper
mobile app and do it right.
0:33:15 But yeah, having to, to design and
develop two separate UIs for two different
0:33:19 use cases was just, was just awful.
0:33:21 I mean, just time time wise, it was
just a bad decision, but I did, I
0:33:24 did get the mobile app working and I
released it like, and it was a used
0:33:27 thing that most people use, I had
like, I don't know, 900 installs on,
0:33:31 on iOS and, uh, some, a couple of
hundred on Android, I think as well,
0:33:35 people use it and it, it had the same.
0:33:37 So like, React Native
powered the UI part of it.
0:33:41 Um, and then in the backend, the syncing
engine and all the exact same code ran.
0:33:44 And so the syncing, the syncing stuff
would, would run and it would send
0:33:47 messages off to the server and then
it would be synced back to the desktop
0:33:50 and, and it worked pretty well.
0:33:51 That must've been a magical moment.
0:33:53 Once you had your data from the desktop
app show up in the mobile app, you make
0:33:56 changes on the mobile app and things
are appearing on your desktop app.
0:34:00 All that work paying off.
0:34:02 Yeah, yeah, I think I have a, I might have
a tweet, I think that showed them side by
0:34:08 side and it was like, look, you can change
one thing here and it shows up over there.
0:34:11 And it's always a cool,
like a cool demo able thing.
0:34:13 Yeah, I think if as like local-first apps
are luckily becoming more and more normal,
0:34:20 I think we will take it for granted.
0:34:22 But I feel quite nostalgic about like
the, the days where this is not normal.
0:34:27 And like the, the magic is really strong.
0:34:29 And in my opinion, it's still
strong when you see like the, the
0:34:33 real time collaboration of apps, et
cetera, and those things just work.
0:34:37 So you've mentioned that you've
been using what was called Node.
0:34:41 js mobile.
0:34:42 And so that allowed you to
bring most of your code there.
0:34:45 Did even like the, the SQLite bindings
and the SQLite reactivity system, did all
0:34:52 React Native
0:34:52 It didn't just work, but it
wasn't too hard to get it to work.
0:34:55 It was sort of just like Electron.
0:34:57 I had to figure out how to load in.
0:34:59 I think there was already an
Electron like C library that
0:35:02 allowed me to easily access SQLite.
0:35:05 But for React Native, I actually
built, I think I built my own SQLite.
0:35:08 Bindings for react native.
0:35:12 I'm actually trying to remember,
maybe I, maybe I did it.
0:35:14 Maybe it did just work because
the, no, I think it was weird.
0:35:17 I think the Node.
0:35:18 js mobile allowed you to
compile C dependencies, but
0:35:22 like it didn't fully work.
0:35:23 And so I had to kind
of get things working.
0:35:25 So it didn't just work, but it wasn't
like super hard to get working.
0:35:28 You basically had to compile SQLite for.
0:35:31 IOS and Android, right?
0:35:32 And so you have to like hook in their
build process and get that loaded.
0:35:35 And Node.
0:35:36 js's mobile support for like C
dependencies from Node wasn't straight
0:35:41 out of the box, but honestly it
worked, it worked well enough to where
0:35:44 it wasn't that hard to get working.
0:35:45 So I was, I was pretty impressed
by that, but I did, I did have
0:35:48 to kind of wire up the, the, the
core things that I still needed.
0:35:51 Got it.
0:35:52 But I suppose that's mostly one of work
and the overall goal of like reusing
0:35:57 the same code from your Actual desktop
app in the mobile app to the most
0:36:01 degree aside from the UI that paid off.
0:36:03 Yes, I think that paid off like
that specific investment worked.
0:36:07 The real downside to all of this stuff
about actually like actually having an
0:36:12 app and especially for Mobile is that
the ecosystem just changes and your app
0:36:17 is in this like proprietary app store.
0:36:19 And like the proprietary app store
is like forced you to, to update it.
0:36:23 Like I think Android at one point forced
me to like, they basically deprecated an
0:36:27 API and I could not release a new version
of the app until I stopped using that
0:36:32 old API and started using like a new one.
0:36:34 The new one requires some really
weird thing that like triggered
0:36:37 a fault, like a crash on Node.
0:36:39 js mobile.
0:36:40 So like, that's a super risky.
0:36:42 Place to be in and so it, it paid
off, but like, it was a continuous
0:36:46 investment risk that didn't require
any maintenance really, except when
0:36:52 I was forced to buy the ecosystem.
0:36:54 And so that's the real
win of the web, right?
0:36:55 Like, you're not like, the
web definitely is weird.
0:36:58 sucks in a lot of ways and like
they change things and they
0:37:00 force you into certain things.
0:37:02 But generally speaking,
backwards compatibility is a
0:37:04 huge, huge value on the web.
0:37:06 And it's just not the case
in the mobile ecosystem.
0:37:08 And they have leverage on you
because you have to go through
0:37:11 their app stores, which just, I
just hate, like, I love mobile apps.
0:37:15 I love having like a real
app, but like that Node.
0:37:18 js mobile was like a risk for sure.
0:37:21 Because There's like certificates and how
things are assigned that kept changing.
0:37:25 And like was, I luckily
never got totally stuck.
0:37:29 Andre actually, I think was the
only other person using this.
0:37:31 And so luckily he had
like a PR that fixed it.
0:37:34 Cause he, he was stuck in the same way,
but he actually knows how to change stuff.
0:37:37 And so he had a PR merge like
three days earlier that luckily
0:37:41 I was able to update Node.
0:37:42 js mobile and it finally
worked, but it's scary, right.
0:37:45 To be.
0:37:45 Like I could have just been totally
stuck as far as I know, there's not
0:37:49 a great way to have that set up today
without a similar level of risk to,
0:37:53 to, because you're, you would have to
build a pretty bespoke sort of custom
0:37:57 native app on, on mobile to, to do this
whole, uh, truly local-first type thing.
0:38:02 Got it.
0:38:03 Yeah.
0:38:03 I think, well, as you mentioned, uh,
well, most platforms are evolving over
0:38:08 time, some more aggressively and yeah,
with little tolerance for developers
0:38:12 who don't update the apps, I think the
web is more graceful in that regard.
0:38:16 On mobile, the rock is sometimes
being pulled underneath you.
0:38:20 But on the, seeing the glass half
full, you also get hopefully improved
0:38:24 APIs and the ecosystem is improving.
0:38:27 I've been recently getting a bit closer to
the React native ecosystem and Expo seems
0:38:32 to make a lot of that quite a lot nicer.
0:38:35 So there's an impressive amount of.
0:38:37 Bindings to native APIs that
you, that you might need.
0:38:40 So, and also given that JavaScript
is also evolving in terms of
0:38:44 standards, supported standards.
0:38:46 Uh, I think there's now also the
lines are getting a bit blurrier
0:38:50 of what needs to run a Node.
0:38:51 js or what is just supported
by like other JavaScript
0:38:55 execution environments or mobile.
0:38:57 You have JavaScript
core on iOS on Android.
0:38:59 I don't know what it's called, but
you also have Hermes, the tool by the
0:39:03 JavaScript runtime by Facebook, I think,
which is specifically designed for mobile.
0:39:08 So I think you get more options,
but there's still like sharp edges.
0:39:13 So did you eventually give up
on those mobile apps or did
0:39:17 you, did this lead you to web?
0:39:19 How, how did the journey
continue from here?
0:39:22 Working on mobile apps
0:39:22 Sure.
0:39:22 So I'd never gave up on the mobile
apps until I fully open source Actual
0:39:26 when I kind of gave up on Actual and
entirely as, as a whole business.
0:39:29 And that was when I told the
community, Hey, if you want to,
0:39:33 like, this is the source code, I
included the source code of it.
0:39:36 And so if you want to figure out how to
like build this and purchase a developer
0:39:40 account and get this working, like it's.
0:39:42 Totally up to you.
0:39:42 But in terms of the Actual apps
that were on people's phones at the
0:39:46 time, yeah, there was never going
to be another update of those.
0:39:48 I'm still working on shutting down Actual.
0:39:50 And so those apps still do exist in the
App Store as far as I know, but they
0:39:53 will be removed when I shut down Actual.
0:39:54 So there was another phase where
I had not given up yet, but I did
0:39:59 sort of admit defeat in terms of
Where the investment should go.
0:40:03 And I just was like, I
need to be on the web.
0:40:06 Like at that, there was a
point when Actual just like,
0:40:08 didn't even work on the web.
0:40:09 It just was not a thing.
0:40:10 And so I think I might have
had a demo on the web, but it
0:40:13 just like, didn't support it.
0:40:14 Loaded the loaded in the
entire SQLite file locally.
0:40:18 And then it just like,
didn't persist anything.
0:40:20 Right.
0:40:20 It just like the app ran, but it didn't
actually, it kind of like removed a
0:40:24 lot of the functionality and it was
just like a quick demo, but like the.
0:40:27 Man forcing people to, to, to
download an app and running into
0:40:31 problems with their local device.
0:40:32 Like it just, the, the web is such
a powerful distribution mechanism.
0:40:36 And I think that's, you know, everybody
knows that and it's really hard to, to
0:40:40 fight against to fight against that and
to force people to, to download apps.
0:40:44 I honestly still love apps.
0:40:45 Like once it gets set up,
I think it's really nice.
0:40:48 Like if it's an app that I use almost
every day, or even just a couple,
0:40:51 like a couple times a week to have it.
0:40:52 In my, in my doc and I can like close it.
0:40:54 I can quit it.
0:40:55 It doesn't have to be in my mess of tabs.
0:40:57 I have a billion tabs and I, so
like managing my, an app that I use
0:41:01 frequently in the browser to me, I,
I don't really like that, but at the
0:41:05 other, at the same time, There are
things that I use all the time on the
0:41:09 web that are only in my tabs, actually.
0:41:11 And it's very kind of nice to
like be able to close the tab and
0:41:14 then like quickly open up a tab
and go to the app really fast.
0:41:17 Whereas like the app startup for,
you know, Mac OS tends to be like
0:41:20 at least two or three seconds.
0:41:22 So I don't know, it's, it's weird.
0:41:23 I kind of say that I like local
apps and yet I live in a browser
0:41:26 like 80 percent of my time.
0:41:28 So there's something there.
0:41:30 And as much as web technology
kind of sucks, and it's so
0:41:32 confusing and I don't like it,
the web is a really powerful draw.
0:41:36 And so I eventually was just like, you
know what, I need to figure this out.
0:41:39 So I never like stopped investing
in the Electron and, and Node.
0:41:43 js or the um, mobile Node.
0:41:45 js mobile stuff.
0:41:46 I would still update.
0:41:48 Do a release of those
every couple of weeks.
0:41:50 But the first, I remember the first time
that I got Actual working on, um, on the
0:41:54 web and I used, you know, that's where
absurd-sql came from, where I was able
0:41:58 to, to compile things out at a compiled
SQL light to web assembly and use
0:42:03 techniques such that like you can open
the app in multiple tabs and actually
0:42:07 change the data, actually query the data.
0:42:09 And like, when you change data
on one tab, that when you.
0:42:12 Query that data on another
tab, it actually shows up.
0:42:15 I got all that working, which is
really, really great, which is
0:42:17 something that we can talk about more.
0:42:19 I remember like the first time that
I released that and people were
0:42:22 just like instantly able to use it.
0:42:24 And then when I was able to fix a bug
and I was just like with like, basically,
0:42:28 basically in, in, in our sync command.
0:42:31 Was able to like, get that bug in
people's hands or that, that, that
0:42:35 bug fix, um, out there was just like
insanely addicting that, that, I mean,
0:42:39 that's the power of the web, because also
actually, it's just a local app, right?
0:42:44 That's the other thing that I
realized when I built the web app, it
0:42:47 literally is a set of static files.
0:42:49 There's no.
0:42:50 Like there's a syncing service needed,
but like in terms of the entirety of
0:42:54 the app itself, I don't need to do
any database mutations when I deploy.
0:42:58 There's no, there's no large CI pipeline
that does all of these complicated
0:43:02 things to deploy, to restart services, to
update services and do all of this stuff.
0:43:06 It is literally an HTML file and
like five JavaScript bundles.
0:43:11 I can just rsync those over to my web
server that host static files and users
0:43:18 like load those new static files and then
it queries their local data and all of
0:43:21 their local stuff is now reading that
like fresh stuff, but like, just the
0:43:26 ability to just like sync those files
over and just, you know, Deploy like
0:43:29 a hundred times a day just unlocks an
iterative development speed that just
0:43:34 cannot be matched by mobile development.
0:43:36 Yeah.
0:43:37 Waiting for the iOS app store to
finally approve your app to be
0:43:40 released is not a fun state to be in.
0:43:43 And having the ability to just in
a matter of seconds, release a new
0:43:47 version of the, of the web app.
0:43:49 It's just so liberating.
0:43:51 So you've now went from this transition
of initially building an Electron
0:43:55 app, which was just an Electron app.
0:43:57 And then you went to another
platform, mobile, Android, and iOS,
0:44:02 where you could still bring Node.
0:44:04 js with you.
0:44:05 And so the, the platforms were still
like similar enough and they, you could
0:44:10 leverage the native aspects of those
platforms still to the extent that
0:44:15 you've so far needed, um, uh, But the
web is quite different in that regard.
0:44:20 There is no native C
bindings you can leverage.
0:44:24 And I'm not sure how far Wasm was along at
that point, but there would have been one
0:44:29 path where we just say, okay, I need to
completely rewrite Actual, maybe even give
0:44:34 up on SQLite and just like embrace all the
standard things that people do on the web
0:44:40 or somehow bring the architecture and the
technological choices that you had so far.
0:44:45 Bring them to the web.
0:44:47 And that means like a
pretty intense pioneer path.
0:44:50 And I think you've chosen the letter.
0:44:54 Bringing the app to the web
0:44:54 Absolutely.
0:44:55 Yeah, it was really a fun, a fun
experiment because the things
0:44:58 that I was using were not that.
0:45:01 Like novel SQLite was really like
the biggest one that just did not
0:45:04 work on the web, everything else.
0:45:06 You know, like if you need like
a background process, you can
0:45:08 just fire up a, a web worker.
0:45:09 It's, it's not too bad.
0:45:11 I'm trying to think of other things I did.
0:45:12 So I was using the, the old node async
hooks, which is now the async local
0:45:17 storage for some really, really neat
stuff for, for the undo and redo mechanism
0:45:22 for Actual, uh, it actually tracks the
messages that get generated for an entire.
0:45:28 Like API requests.
0:45:29 So I have these like handlers.
0:45:31 And so it sets in local and async
local storage, like a buffer of
0:45:35 messages that is fresh each time.
0:45:38 And so when you send an action to do,
while it's executing that entire, like
0:45:43 transaction create method, it tracks
all of the messages that are done
0:45:47 during, like created during that time.
0:45:49 And then it, and then when that
action is done, it reads that from
0:45:53 that buffer and then it packages
them up as like an, like an undo.
0:45:57 Packet, right?
0:45:58 That, and that then gets
stored onto another queue.
0:46:00 And so if you want to undo that,
that's how it knows like which
0:46:04 messages and through which points
of time it needs to like, it needs
0:46:08 to undo up to a certain time, right?
0:46:10 And so the async local source is
really, really great for that.
0:46:12 Cause I didn't have to thread
through all of this stuff.
0:46:15 The web just doesn't support
those kinds of things at all.
0:46:17 So there are things like
that, that I had to undo.
0:46:20 Luckily, that actually improved things
on mobile as well, because Node.
0:46:23 js mobile was actually, I was
finding some, some bugs with some
0:46:27 of those more advanced Node APIs.
0:46:30 So there are things that I had to
simplify and like kind of undo,
0:46:33 but for the most part, SQLite
was like the main hard problem.
0:46:37 And so yes, this is where I, I
experimented and I played and I
0:46:41 was like, it's I mean, WebAssembly
was pretty mature at this point.
0:46:44 I think this was like 2020 or so.
0:46:46 Um, I could be wrong on that, but it
was, you know, like WebAssembly was
0:46:49 like a pretty good at that point.
0:46:50 I had no problems like trying
to bet on WebAssembly then.
0:46:54 It was not that hard to compile.
0:46:56 Uh, SQLite's WebAssembly
at that time, the SQL.
0:46:58 js was, I think, the big library that
already came with the pre compiled SQLite.
0:47:01 So I got the app running with SQL.
0:47:04 js and without a ton
of work, to be honest.
0:47:07 Then it was like, crap, what
happens when you change the data?
0:47:10 Because SQLite literally just slurps in
the whole database in local memory, right?
0:47:15 So you can change the data.
0:47:16 You can be working with your data.
0:47:17 I mean, it all works totally fine.
0:47:20 But then you refresh the tab
and then all that data is lost.
0:47:22 So obviously that's not,
that's not going to work.
0:47:24 So I came up with all these ideas,
like, well, what if I like stored the
0:47:28 messages persistently in the background?
0:47:31 And then like, when you
refresh the tab, it.
0:47:34 You know, reapply this
messages after loading it up.
0:47:37 But that means that every single time
you open up a new tab, it's loading
0:47:40 in the entire database into memory.
0:47:41 And like, for me, at least that
was like a hard requirement.
0:47:46 Do not load the whole
database into memory.
0:47:48 Even worse than that, do not write
the whole database back into memory.
0:47:52 Cause that was, that was the thing that
some people were doing at this time.
0:47:55 They were like, Oh, just
solve this persistent thing.
0:47:57 Okay.
0:47:57 You change one number from four to
five, and you're going to rewrite
0:48:00 this entire six megabyte database.
0:48:03 Back into memory, it is so inefficient.
0:48:06 And I think that's one of the, my gripes
with the web is that like, we've, we've
0:48:09 lost the plot of software development
where it's, where our bar for, for quality
0:48:16 and, and good engineering practices
is so low and it's, Might be a spicy
0:48:22 take, but like, we just accept the
fact that things have to be this bad.
0:48:27 And like, it's so like acceptable to be
like, well, there's nothing I can do.
0:48:31 It's out of my control.
0:48:33 Right.
0:48:33 So I'm just going to write, it's right.
0:48:34 Six megabytes of memory.
0:48:35 Every single time I change
something that is like so bad for
0:48:39 like your computer's hard drive.
0:48:41 It's so bad for your.
0:48:42 Memory consumption, your, your power
is so bad in all sorts of ways, right?
0:48:45 It's so wasteful.
0:48:46 Man, it just makes the app entirely
slow and prone to all kinds of problems.
0:48:51 Like it's, I do have gripes with
just, I feel like our bar as
0:48:55 an industry needs to be higher.
0:48:56 And I, I honestly think that web
development is to blame for some of that.
0:49:00 That's my spicy take because it makes
it easy for us to throw up our hands.
0:49:03 And say, I can't go past
that layer of abstraction.
0:49:07 Whereas in the native
world, you have a problem.
0:49:09 You can dig into those, the C binaries.
0:49:13 it might be hard, but you have the
power to go in and change things.
0:49:17 Whereas on the web, it's
like, it just works that way.
0:49:19 It sucks.
0:49:20 Like too bad.
0:49:21 I think we need aim, aim higher.
0:49:22 And so that's, that's, that
was my approach, right?
0:49:25 I was like, I'm not, I'm
not going to accept this.
0:49:26 This is unacceptable.
0:49:27 Is there a way I, um,
I can get this to work?
0:49:30 And maybe there wasn't.
0:49:30 And then I would have
to just sort of give up.
0:49:32 But I discovered a kind of
novel thing that ended up
0:49:34 working actually pretty well.
0:49:36 And that's how absurd-sql came
about was I, I figured out a way.
0:49:39 The real trick here was, so don't
read everything from memory, right?
0:49:44 So how do I not read
everything from memory?
0:49:45 Well, I have to store it in an existing
database that exists on the web.
0:49:49 There's no question at the time
that that was IndexedDB, right?
0:49:52 That was like the only
mature database abstraction.
0:49:55 Okay, so I can store all this
in IndexedDB, this whole blob.
0:49:59 But I'm still having to read
this whole thing into memory.
0:50:02 How do I stop that?
0:50:04 So I figured out, I looked
into SQLite's internal APIs.
0:50:06 At this point, I was pretty familiar
with SQLite's internal APIs.
0:50:09 Honestly, it's a really fun read.
0:50:11 They're well documented,
really straightforward C code.
0:50:14 It's really, really fun.
0:50:15 So basically, SQLite works in blocks.
0:50:17 So you have like, I
forget what size it is.
0:50:20 Is it 5k blocks?
0:50:21 Like you can actually change, change the
block size that SQLite works, works by.
0:50:24 But like, I think the default is 5k.
0:50:27 I think like 4k, or 8k,
something like that.
0:50:30 But like, yeah, that,
that ballpark, I think.
0:50:32 Yeah.
0:50:33 So that's, that's the amount that it
reads from your hard disk per chunk.
0:50:36 Like if it needs to run,
read one little bit.
0:50:39 It reads at least that amount, right?
0:50:41 So it like reads a whole block in
and then it does what it needs to.
0:50:44 And then it writes that block back
in if it, if it needs to write stuff.
0:50:47 So it does, that's how it like, doesn't
read everything in from all the memory.
0:50:51 Um, and so I thought, I was like,
okay, well, what if, what if in
0:50:55 indexedDB we store these blocks?
0:50:56 Like that is the, like, this needs to
be just basically like a file system.
0:51:00 Like I need to make this
treat as a file system.
0:51:02 And so I.
0:51:04 Started playing with this
and I, I, conceptually, I was
0:51:07 like, this, this could work.
0:51:08 Like, I don't see any reason why
this couldn't work because once
0:51:12 I have that, once you can read
different blocks and you don't have
0:51:14 to read the whole thing into memory.
0:51:16 Well, SQLite, it was very
straightforward in SQLite's code
0:51:18 about how it writes stuff down.
0:51:20 I was like, well, I can also, I
can also just write the blocks.
0:51:23 Right?
0:51:23 Every, I feel like every, every
three or four years I have this,
0:51:27 like, I see a gap and like, it's,
it's intriguing to me a lot.
0:51:31 And like, I, I like prettier
was kind of this thing too.
0:51:34 Like I see this thing that's like, man,
there's this problem that everything
0:51:37 is having, but like, I see this thing
that works really well over here.
0:51:40 Like, can we just.
0:51:41 Take that from over there
and like apply it here.
0:51:43 Like, like, does that work?
0:51:44 And you know, a third, a third
of the time, it ends up being an
0:51:47 interesting thing that does work.
0:51:48 Essentially all of that to say,
I discovered how to store these
0:51:53 blocks in, into IndexedDB to read
and write them from, from IndexedDB.
0:51:58 And I was able to leverage
IndexedDB's transactional
0:52:02 semantics so that locking worked.
0:52:05 And that's the real critical thing here
is that like SQLite, heavily relies
0:52:09 on file system locking API, such that
says, I'm going to lock this block.
0:52:13 I'm going to lock this piece of data here.
0:52:15 Do not let anything else write to that.
0:52:16 That is a stable database.
0:52:18 Like that is how it does not get corrupt.
0:52:19 If it, if you break those
semantics, it will get corrupt.
0:52:22 And so, um, I was able to leverage.
0:52:25 IndexedDB is like locking semantics.
0:52:27 And I was like, I can map these
onto these such that like, when I'm
0:52:30 writing this thing, nothing else
should be able to read from it.
0:52:32 And it worked.
0:52:33 So you can like load in
the data in different tabs.
0:52:36 You can write to that in like one tab
and like see that data in the other tab.
0:52:40 And it worked out really well.
0:52:41 And I was like, this would be
a really interesting thing to
0:52:43 open source and talk about.
0:52:43 And so.
0:52:44 It became absurd-sql and it became
like a, a pretty influential thing.
0:52:48 I think, honestly, this kind of project,
I invest like a month or two of my life
0:52:52 into, and then like, I have a day job.
0:52:54 And so I just did not have a lot of time.
0:52:55 Like I got it working for Actual and it
deployed Actual and it worked on the web.
0:52:59 And that was amazing, but I just didn't
have the resources or the time to really
0:53:03 like build it out or like fix really
obscure bugs that people were coming up, I
0:53:07 sort of like kind of left it as a project
that wasn't, that wasn't Maintained
0:53:10 super well, and sometimes I feel guilty
about that, but I think there's other
0:53:13 community members that have taken it on.
0:53:14 It's, it's more of like an
influential prototype, I think,
0:53:19 Absurd-SQL
0:53:19 Yeah, I think for me, this is
actually kind of like a, a definition
0:53:24 of a kind of project category.
0:53:26 Like not every project needs
to like prettier lived on and
0:53:30 now it's like super common.
0:53:32 And like, I think that's one very
successful way how a project can evolve.
0:53:37 But I think with absurd-sql, you've built
something it's even more than a proof of
0:53:41 concept because you used it in production.
0:53:44 I, I use it for a while in production.
0:53:46 Other people use it for a while in
production, but, uh, like you said, like
0:53:50 it became a very influential project.
0:53:53 And I think it showed to the world,
like, Hey, we don't need to settle for
0:53:57 saving six megabytes or 50 megabytes
for every little change as you do.
0:54:02 Want to do persistence in the web before
people would just like ride, like.
0:54:07 80 megabyte JSON files, like all the
time to, to index to be, and then we're
0:54:12 then wonder why their, the app is slow
and their, their CPU is going crazy.
0:54:16 So you aiming for higher, I
think that has a massive impact.
0:54:20 And I think that's now those
ideas are still like leveraged
0:54:24 now in other projects.
0:54:25 Whether I think.
0:54:26 The wa-sqlite, um, project that is,
I think also using some, some similar
0:54:32 approaches and by now we also have a
different storage mechanism and the web
0:54:37 that is more and more common, namely
OPFS, original private file system.
0:54:43 Which also already gives you a file
system representation where you can
0:54:47 lock individual files, et cetera.
0:54:49 I think the details are still being
figured out right now, but that's, for
0:54:53 example, what I'm using for Overtone and
in production, that's already quite nice.
0:54:57 But I think you've really, you, you went
super early on that and very daring from
0:55:03 first principles and that's so admirable
and that, that like very inspiring.
0:55:07 Thank you very much.
0:55:07 I appreciate that.
0:55:08 Sometimes I, I don't know.
0:55:10 I, I don't really track.
0:55:12 The, like some of the fallout sometimes,
like I, I just had to invest in my job
0:55:16 a lot for the next like six months.
0:55:18 And so sometimes I'm, I'm not sure.
0:55:20 I don't, I don't know how much influence
it has, but it's, it's good to hear
0:55:22 feedback that work is impactful.
0:55:25 I, I would love to hear about your
experience with, with it, with OPFS.
0:55:29 And, and, uh, are you using
the file system native access
0:55:31 stuff as well and in your app?
0:55:33 How's that going?
0:55:34 The latter, not yet.
0:55:35 I'm also planning to, since like
for Overtone, which is a music
0:55:39 player, I do want to also support
you bringing your own music that
0:55:43 you might have like in a download
folder on a music folder somewhere.
0:55:47 Right now I don't use that
particular browser APIs yet.
0:55:51 But I'm using OPFS for many purposes.
0:55:54 I'm using OPFS for persistence
of a SQLite database, where you
0:55:59 would have used IndexedDB before.
0:56:01 And I think some people still
use IndexedDB for targeting
0:56:05 older browsers as well.
0:56:06 But I'm also using OPFS for like what
you would use a file system for as
0:56:11 well, which is like storing files.
0:56:13 So for example, Before displaying
images in the app, I actually don't
0:56:18 use like just an image tag and
then point to an external URL since
0:56:22 those URLs might go away, et cetera.
0:56:24 So I actually download those images, store
them in OPFS and then pre process them.
0:56:30 Um, on a worker and sent them
over on a, on an off screen
0:56:34 canvas to the main thread.
0:56:35 So I'm like using some, some more
like native development practices here
0:56:40 and trying to bring them to the web.
0:56:41 Very, very much sharing the same
opinion, like your spicy take from,
0:56:46 for me, not so spicy take that
we should aim higher in the web.
0:56:49 And I think we can learn quite a lot from.
0:56:52 More native development backgrounds.
0:56:54 And this is, I think this is how we
get, uh, after all pretty fast web apps.
0:57:00 Figma is another notable example
there where how you get actually a
0:57:04 really high performance app that feels
nice is by aiming higher and, uh,
0:57:10 bringing some of those methodologies
from other environments to the web.
0:57:14 So I think.
0:57:16 There's still a couple of
interesting aspects in absurd-sql
0:57:19 that we haven't yet gotten into.
0:57:21 So you've mentioned that you're
using index to be, uh, with, uh, the
0:57:26 block, um, the, the block storage
and the index to be transactions,
0:57:30 but to actually write that you could.
0:57:33 I guess either do, could you do
that on the main thread since you've
0:57:37 chosen to do that on a worker?
0:57:38 So maybe you can talk a little bit
about the, the threading model and
0:57:42 also how you even went beyond IndexedDB
transactions and using atomics and
0:57:49 shared array buffers to still slay some
dragons that needed to be slayed here.
0:57:54 Yeah, there is one little trick that I
discovered and I think I was actually
0:57:59 a live stream and I think I have a
recording of it, which was kind of fun.
0:58:01 It's like a fun little blew
my mind at, um, at the time.
0:58:04 And I think this is a trick
that was independently
0:58:07 discovered by several people.
0:58:08 I think there's a, uh, there's another
library that I think from is from some,
0:58:12 some Google folks about how, like,
like loading in third party things
0:58:15 on, you know, In like an iframe or
something to kind of like sandbox things
0:58:19 that that uses this same technique.
0:58:21 But essentially here's,
here's the problem.
0:58:24 When you compile SQLite through
WebAssembly, you can intercept
0:58:28 the API calls that it makes.
0:58:30 So you can intercept the like read
The read and write command, like
0:58:33 there's a C API is reading right
for reading and writing files.
0:58:37 And so you can say, okay, I'm going to
implement the read command for like,
0:58:41 through WebAssembly, the WebAssembly
compiled version of this SQLite.
0:58:45 I'm going to implement the
read command and like read some
0:58:47 data for it and give it back.
0:58:48 So I can just read from index, from index
db and everything is going to work right.
0:58:52 The, the problem is, is that
the read command from C is
0:58:55 completely synchronous, right?
0:58:56 So the WebAssembly compiled binary
expects that to be synchronous.
0:59:00 Like you can, it'll call out to
JavaScript and you can implement the read
0:59:03 command, but you cannot await in there.
0:59:06 Like you literally have to, in that event
loop, synchronously return some data.
0:59:10 And so, uh, when I hit that, I was
like, I thought I was dead in the water.
0:59:13 I was like this, this.
0:59:14 I just can't do this.
0:59:15 Maybe, maybe it wasn't that bad
actually, because I think I did
0:59:18 see, so one option is you can in
WebAssembly, I think it has a way to
0:59:22 convert synchronous APIs to async.
0:59:24 I think it's called like
asynchronify or something like that.
0:59:27 I was terrified of it to be honest.
0:59:29 And so I thought it just
wasn't going to work.
0:59:30 I was terrified about the performance.
0:59:32 I've had experiences in some
codebases that overuse of asynchronous
0:59:36 APIs really killed performance
because waiting one promise tick.
0:59:41 In every single layer of the
abstraction is very not noticeable
0:59:46 at a small scale, but it very
becomes noticeable at a large scale.
0:59:50 And the really pernicious thing about it
is that it's death by not a thousand cuts.
0:59:54 It's death by a million cuts.
0:59:55 Like, it's, you can't
go in there and remove.
0:59:58 20 by 20 awaits, right?
0:59:59 And like, it'll be fine.
1:00:00 The thing that I don't like about
async code is that as you abstract
1:00:03 things away, so if you take it from
two abstractions to 10 abstract, like
1:00:06 you split a two async functions into
10 async functions, well, every one
1:00:10 of them has to await on each other.
1:00:12 And so it's literally making it slower.
1:00:14 And that's not the
case , with synchronous stuff.
1:00:16 You can make it two functions.
1:00:17 You can make it 500 functions.
1:00:19 You can abstract things
the way that you want to.
1:00:21 The shape of your abstraction
literally impacts performance when
1:00:25 you're in, in, um, async world.
1:00:26 So I have experience with code bases that
like, that did things in a really weird
1:00:31 way, like awaited literally everything.
1:00:33 And it was, it was causing
all sorts of perf problems.
1:00:36 And so that, For that reason, from my
experience, I was terrified of it because
1:00:40 I thought it made every single function
in C in the WebAssembly binary async.
1:00:44 It turns out that I was probably wrong
and I probably should have done my
1:00:48 own investigation in like performance
profiling because I got some feedback.
1:00:52 Later on, like months after I released
absurd-sql, I think from some people
1:00:56 that I respected pretty well that
were saying, uh, the async stuff is
1:01:00 actually fine, like it actually has
a very negligible performance impact.
1:01:04 So there's probably some tricks
that they, they, they do that.
1:01:07 So it's probably fine.
1:01:07 And I think that WA SQLite
project that you mentioned, I
1:01:11 think it actually uses this mode.
1:01:14 And the good thing about that is
that it doesn't need atomics, which
1:01:17 means it doesn't require HTTPS.
1:01:19 So I'm probably.
1:01:21 Let me jump back a little bit to
explain why atomics are even needed.
1:01:24 So let's go back to that
synchronous method, right?
1:01:26 So I need to call into IndexedDB
and get and do an async API.
1:01:30 How am I going to do that?
1:01:31 Well, there's a little trick that
you can do by making an async call.
1:01:37 And I believe that it's Has
to be on a different thread.
1:01:42 So there's like two
background threads, right?
1:01:44 There's like the normal backend thread.
1:01:46 And then there's another separate thread.
1:01:48 That's like the thread that reads and
writes from index CB like asynchronously.
1:01:51 Right.
1:01:52 Between those two threads, you
share a shared array buffer, which
1:01:55 is this really low level, really
interesting thing on the web, which
1:01:59 is shared memory across threads.
1:02:01 And on top of that, you, uh, there's
APIs you can use called atomics, which
1:02:05 allow you to interact with this shared
array buffer, and you can actually
1:02:08 coordinate across the threads and do
some really, really powerful things.
1:02:12 So one of those things that you can
do is you can write to that shared
1:02:15 array buffer in a certain way.
1:02:17 And then in another thread,
you can call atomics.Wait.
1:02:20 And what atomics.wait does is it literally
synchronously blocks that thread from
1:02:24 running like it, you call that thread.
1:02:26 And if it does not wake up,
then it won't wake up ever.
1:02:29 Like you call atomic weight and you
tell it, wait for this bit to be
1:02:33 flipped in the shared array buffer.
1:02:34 And then the other thread can
flip that bit when it's not.
1:02:37 done, and then the other
thing can continue executing.
1:02:40 Using that technique, we can read from an
asynchronous thing in the second thread,
1:02:44 do whatever we need to, and store that
data in some sort of buffer somewhere.
1:02:49 And then while the first thread is
blocked on that atomic set, wait.
1:02:52 And then once the, once that
bit is flipped and it continues
1:02:55 executing, then it can read from that
buffer and actually get the data.
1:02:58 And so using that technique, Um, I
use that technique in every single API
1:03:02 in, in Upstairs SQL to turn an async
function into a synchronous function.
1:03:06 And that's how it can
interface with IndexedDB.
1:03:09 The downside though, is that to use
atomic set weight, it's one of those
1:03:13 newer APIs that Chrome and other browsers
force HTTPS a secure context on you.
1:03:18 And so your app has to be running under
HTTPS, which I've always said, like.
1:03:22 Who cares?
1:03:23 Like, of course you're going to
be running under that anyway.
1:03:25 It's been enough of a,
kind of a pain point.
1:03:27 Like people are using like weird
reverse proxies or doing their
1:03:30 own things where suppose, I guess
some people just don't really care
1:03:32 and they just want to run HTTP.
1:03:33 And so Actual, like the open source
version of Actual can't run under HTTP.
1:03:37 And that's like, you always
have to be setting up to me.
1:03:40 It's, I don't know, I still am a
little bit like, I don't really care.
1:03:42 Like you can just like set up
HTTPS, but it's enough of a source
1:03:45 of a pain that I can see benefits
and not, not having to require it.
1:03:49 Right.
1:03:49 Yeah, I think this was also related
to the, uh, Spectre exploit at
1:03:53 some point, uh, vulnerability.
1:03:55 And I think it's not just that you need
to run on HTTPS by now, but I think you
1:03:59 also need to have a few HTTP headers set.
1:04:03 I think like cross origin, uh,
open policy and embedders policy.
1:04:07 And I think there was also like
a limitation that Safari didn't
1:04:10 support it well for, for some time.
1:04:12 So yeah, there it's, Still, browsers
are still growing up, but I think at
1:04:17 some point this can be assumed that, uh,
this will just work, but on the other
1:04:21 side, I think those tricks might also,
you mentioned that asyncify approach.
1:04:26 So that is also another option.
1:04:28 And I think there's even a new
approach stabilizing right now.
1:04:32 Where WebAssembly natively
can integrate with Promises.
1:04:36 So I think that's a, that's a new, um,
development around WASM and browsers.
1:04:42 So I think there's, you, you certainly
use quite the bleeding edge there.
1:04:47 And so, yeah, right now it's
getting more stabilized.
1:04:51 But I do think there is some truth
to what you've mentioned in regards
1:04:55 to avoiding asynchronous code when
possible, since it's not just like
1:04:59 a potential performance overhead.
1:05:02 So, and I think some people go even
as far as saying that going from
1:05:06 callbacks to async await and promises.
1:05:09 Was one of the biggest
mistakes in JavaScript.
1:05:12 I think that can also be
considered a hot take.
1:05:14 Let's see where we'll end up
in a couple of years on that.
1:05:17 But, uh, aside from the performance,
I think another common downside
1:05:22 of asynchronous code is that it
basically introduces distributed
1:05:25 systems problems into, into your code.
1:05:28 In this case, where we basically
just wrap an API, I think it's okay.
1:05:32 Um, But, uh, that's a, that's another
notable difference of like how, what
1:05:37 we've been exploring with LiveStore
and, and Riffle is by really making
1:05:42 the, like in a browser context, still
from the main thread, allowing for
1:05:46 synchronous SQL queries, which return
very fast and therefore like you can just
1:05:52 write your normal JavaScript as you do.
1:05:54 comes at a risk of potentially
blocking the main thread.
1:05:56 That's a, that's a different
challenge, but if you, if you have
1:06:01 performance under control, it gives
you a much simpler programming model.
1:06:04 So that's another weak one.
1:06:05 Yeah, that's, I'm all here
for synchronous SQL queries.
1:06:09 I think it's crazy.
1:06:11 I think there's been no libraries
before that integrated with SQLite 3 and
1:06:14 like made all of the APIs, um, async.
1:06:16 It was crazy.
1:06:17 Like it's such a in
memory super close local.
1:06:20 thing and , you really want at the
low level to, to provide the at
1:06:24 least option to be synchronous.
1:06:25 And it, so that I just remembered
actually, it's not just
1:06:28 wrapping an async function.
1:06:30 So like it's, it's an internal
implementation detail.
1:06:33 If you go the asynchronous by route,
which turns C functions into asynchronous
1:06:37 functions, therefore you can interact
with like asynchronous stuff.
1:06:41 Believe that that, that forces you
when you call a SQLite by like method,
1:06:47 that method is an asynchronous method.
1:06:49 You cannot synchronously execute a SQL.
1:06:51 That was a big reason why I also really
did not want to use WI SQL because I
1:06:56 wanted my functions to be synchronous.
1:06:58 My app depends on many of them being
synchronous and just my workflows.
1:07:01 And it just, it just greatly
simplifies the entire workflow.
1:07:04 There's no reason for
me to make this async.
1:07:06 This is a single client.
1:07:08 App, there's one request
coming through at a time.
1:07:11 I can control it entirely, right?
1:07:13 This is not a web server handling
thousands of requests at a time
1:07:17 to take on the complexities of
asynchronous code with the performance
1:07:21 hits was just ridiculous to me.
1:07:23 And so I, This technique I still think
has has merit, actually, the more I think
1:07:28 about it, and it's something that I owe
this community like a great deal of, of
1:07:34 like blog posts and and and research.
1:07:36 I need to sit down and really
like go through what the latest
1:07:39 and greatest is and really.
1:07:40 Vetted and see, see what people
are landing on, because if it's
1:07:44 still not possible to do that,
I think that's a huge downside.
1:07:47 I do.
1:07:47 I did just see that the, the
file system access APIs, I
1:07:51 believe they finally converted.
1:07:53 Originally there were some of them
in the, in a worker thread that
1:07:56 were supposed to be synchronous, but
in the spec, we're actually async.
1:08:00 And that made me really, really not happy.
1:08:03 And I filed like a GitHub ticket.
1:08:05 I think they finally, if I.
1:08:07 Just checked earlier this morning.
1:08:08 They finally have on the MDN
page says that they are now
1:08:11 synchronous, which is great.
1:08:12 So hopefully we're moving
in the right direction.
1:08:14 But I think it's, yeah, I think it's
really, I'm all about synchronous 'cause
1:08:17 it means that you can use them in context
that like are synchronous, like there you
1:08:21 might be removing a lot of functionality
because once something is async, the thing
1:08:27 that uses it has to be async as well.
1:08:29 And a lot of these cases for local-first
apps especially, it just is, there's,
1:08:32 there's literally no benefit to it.
1:08:33 , it's a local app.
1:08:35 There's not a second user that
can come be querying this.
1:08:38 I don't know.
1:08:38 I think I get it.
1:08:40 I think we've, we've over
indexed a little bit on this.
1:08:42 And I do think that I
forgot about the headers.
1:08:45 That is because like reverse
proxies can like drop those headers
1:08:48 and then like a user is like,
why isn't this app even loading?
1:08:50 So I, I don't know.
1:08:51 It's a trade off.
1:08:52 I still probably lean a little
bit towards it's worth it.
1:08:55 But yeah, I completely agree with, uh,
I think this is the same theme again,
1:09:00 where in web development, we've kind
of gotten so used to some practices.
1:09:06 And I think it's one is, uh,
being efficient and performance
1:09:09 minded, but another is just
like the programming models, how
1:09:13 they've kind of eroded over time.
1:09:15 we need to deal with distributed
systems problems where they're
1:09:18 just completely accidental.
1:09:20 And , we're just so used to, to like
so many things being asynchronous,
1:09:25 which doesn't need to be asynchronous.
1:09:27 We kind of went from callback
hell to async hell in a way.
1:09:30 And in React we use, useEffect for
so many things where we shouldn't.
1:09:35 And is just so wild
that like most of our.
1:09:38 Data interactions are asynchronous,
like in a way it's almost like if
1:09:43 react would not just give you a value
right away, but would give you like a
1:09:48 tuple of like either it's loading or
a value is like we're already halfway
1:09:52 there just in, in such a bad direction.
1:09:55 So I, uh, I think this is kind of a
subtle difference for people to understand
1:10:00 how much synchronous code execution
can simplify your app development.
1:10:05 But I think I'm, I'm preaching
to, to the choir here.
1:10:09 Yeah, sure.
1:10:10 Totally.
1:10:11 I mean, it helps with debugging.
1:10:12 Like if you're stepping over code,
whenever you're hitting like async
1:10:15 code, Chrome tries to do this thing
where it like will step over the await,
1:10:19 but it only works like half the time.
1:10:21 Like it's yeah, it's super annoying.
1:10:23 I do get this needed probably most of
the time, but it makes me sad when I just
1:10:27 see it like applied without any thought.
1:10:29 So after fulfilling your goal with
bringing Actual to the web through
1:10:35 absurd-sql, which I think is like
just to look back, like how much
1:10:39 pioneering work you've really
done to make this app happen.
1:10:42 Like you started this journey before.
1:10:45 The term local-first was there.
1:10:47 You've built one of the first credible
local-first apps and really invented
1:10:51 so many things along the way, like all
by yourself with the help from, from
1:10:56 some of your, your friends, but you,
like you figured out like how to use
1:11:00 SQLite in like even in a web context in
a reactive way, but then also made it
1:11:05 collaborative through your sync engine and
ultimately brought all of this to the web.
1:11:09 That is an impressive journey.
1:11:11 And I think you've been on the journey,
not just building all of this full
1:11:15 time, but you actually had like some,
some, some Actual full time job next
1:11:20 to that, which I think at some point
was just too much, which led you to,
1:11:24 to at some point, uh, hand over the
project to the broader community.
1:11:32 Transition to Open Source
1:11:32 Sure.
1:11:32 Absolutely.
1:11:33 So I, there was about three or four
years when I was doing consulting work.
1:11:37 And that was during that time was around
when I started it in 2017, around 2019,
1:11:43 I think was the year that I full time
just didn't do any consulting work.
1:11:46 And I was like, I'm just going
to give this a try to see if I
1:11:48 can build this out as a business.
1:11:50 And it got moderately successful.
1:11:51 I think I got up to around 800
users, I think at, um, at the height.
1:11:55 And I was only charging 4 a month, which
again, I had no idea what I was doing.
1:11:59 Um, so too low, but like, you
know, like if you do the math,
1:12:03 that's like, that's not terrible.
1:12:05 A lot of people aren't
even able to get to that.
1:12:07 Like that much, like, but it wasn't even
close to becoming like a full time thing.
1:12:10 I honestly look, looking back and
all the experience that I have now
1:12:13 too, I, I, I could have built it out.
1:12:14 And especially if like Mint shut
down now, like there's so many
1:12:17 things that I've missed that I, I
probably could have built it out.
1:12:20 But, um, building a business
is really hard though.
1:12:22 And it's, I think I realized that I
also had a hard time finding people
1:12:27 to kind of like work with me on it.
1:12:29 Like I kept trying to, people were like
sort of interested in helping and which
1:12:33 made this kind of weird dynamic where
There is like four times when I tried to
1:12:37 allow contributors, uh, like without pay,
they just wanted to help, but without,
1:12:42 without paying them, it's just weird.
1:12:43 Like, they would do something for
a week and then be gone for two
1:12:45 weeks and not even like speak to me.
1:12:47 Like, I can't operate like that.
1:12:48 I'm operating a business here.
1:12:49 So I try to like, I try to.
1:12:51 Think about how I could hire a developer,
but I was so busy with everything else.
1:12:56 It just felt, I had no
idea how to filter through.
1:12:58 I don't know, should I use like,
you know, one of those like Upwork
1:13:01 or other things I had no, this just
seemed like so full of spam and, and
1:13:05 people who had no idea what they were
doing that it just was overwhelming.
1:13:08 I never figured out how to hire people.
1:13:10 And that was, that's my biggest regret.
1:13:11 Probably it's not figuring out how
to involve people in a way that
1:13:14 would help share some of the burden.
1:13:15 And so that, it, Would help it a little
bit, be a little bit more sustainable.
1:13:19 And I was just, I was ready to work with
people again, work at a company that
1:13:22 was like, I could grow from grow, how
to like, learn how to lead people and
1:13:25 learn how to, you know, lead teams and
do like product requirements and kind
1:13:29 of focus more on just what I'm good at.
1:13:31 And so ultimately, yeah, I just
wasn't, I wasn't serving Actual well,
1:13:35 I wasn't serving so that I did that.
1:13:37 Well, I did do that for a whole year.
1:13:38 which was, which was good, but
it, it, I wasn't growing enough
1:13:42 for it to be like a real thing.
1:13:43 And so I got a little burned out on
trying to make it like a full time thing.
1:13:48 Um, and so I was talking to a
friend who referred me to Stripe.
1:13:51 And so I got hired at Stripe.
1:13:52 I did try to work on it kind of
like in the mornings for like an
1:13:55 hour or two and then work at Stripe.
1:13:57 But obviously like that, just when I
got more and more involved in Stripe
1:14:00 and doing like really interesting things
there that kind of absorbed my time.
1:14:04 And so two years ago, I just was like,
I'm not serving Actual well, I'm not.
1:14:08 Spending enough time there.
1:14:08 I'm not serving stripe.
1:14:09 Well, cause I'm not spending enough time
or like, it was too tiring to like do
1:14:13 both and it wasn't serving like my family.
1:14:15 I wasn't serving my
own personal interests.
1:14:17 Well, either it's just like too
way, like way too overinvested.
1:14:20 So I decided to, to
not work on it anymore.
1:14:23 And so the choices was either sell
it or open source it, or just.
1:14:26 Shut it down.
1:14:26 I investigated selling it a little bit.
1:14:28 And this is like, again, some of the
downside of, of doing such a novel app is
1:14:31 that, um, I talked to a couple of people.
1:14:33 It's just so clear that nobody
had not much interest in it.
1:14:36 One, one, it just wasn't making enough
money, but two, like even starting
1:14:39 to explain how it worked, that
just, it's not a, you know, People
1:14:43 aren't interested in buying that.
1:14:44 Like they're interested in buying
like the super run of the mill apps
1:14:47 that they, that they can take and
grow from a thousand to 10, 000 users.
1:14:51 So this app was not that app at all.
1:14:53 So selling it was not in the cards
and with the money it was making, it
1:14:56 wasn't going to sell for that much.
1:14:57 So I decided to open source it and
it was a decent amount of work.
1:15:00 It was decent amount of work
for me for like the year after
1:15:02 that too, to help transition.
1:15:03 Doing it.
1:15:04 And sometimes I was like,
should I have just shut it down?
1:15:06 But I'm really glad I did because
now it's running entirely on its own.
1:15:09 Like I, I'm, I'm not even
on, on the discord anymore,
1:15:12 which I probably should be.
1:15:13 Honestly, I was starting to get
involved in it again in January.
1:15:16 And then like a work thing
happened and totally absorbed
1:15:18 my time for the last two months.
1:15:20 So I will, I want to start interacting
with the community again, but there
1:15:23 there's, they seem to be, they
seem to be running it really well.
1:15:26 And so it's, it's been great.
1:15:27 They've.
1:15:28 They've been taking it and
like fixing a lot of bugs.
1:15:30 And like it, there's definitely like a
power of, of, of open source at work.
1:15:35 It's, it's messier.
1:15:36 I honestly think that there's things,
parts of the app that don't look as good.
1:15:39 They're, they're not as clean, not as,
not as polished, not as thoughtful, but.
1:15:44 There's a lot of added functionality,
a lot of stuff that is good and a
1:15:46 lot of stuff that is improved too.
1:15:48 Thank you so much for, for
sharing this entire story.
1:15:51 I mean, I have massive respect to
you of how you've like navigated
1:15:55 this entire journey and takes a lot
of, I think there's not just on a
1:15:59 technological level, there's like doubt.
1:16:01 It's just like, is this possible?
1:16:03 Does this make sense?
1:16:04 But you're like building a
product, you're building a company.
1:16:07 So there's a lot of uncertainty if you're,
if you're not in a full time job where
1:16:11 someone else Takes care of the things
and you have your responsibilities.
1:16:15 Here's like, everything is uncertain.
1:16:17 And so navigating that while
also like having a family and
1:16:21 other responsibilities, that's,
that's a lot of commitment.
1:16:25 So I have a lot of respect for
like, Not just the decisions you've
1:16:29 made, but like also , how thoughtful
you went about those transitions.
1:16:32 Like how much you, like you said, like
you owe the community, like an update.
1:16:37 I don't think you owe the community
anything you've given the community
1:16:39 a lot, but I think all of your
contributions are really well received.
1:16:44 And, and I think highly
valued by a lot of people.
1:16:47 And so I think even if Actual now did
not work out as like that, uh, That
1:16:52 billion dollar startup by itself.
1:16:55 Uh, I think it will actually influence
a lot of those and who knows, maybe
1:16:59 you take another stab at the same thing
at another thing in a similar way.
1:17:04 And I can't wait to hear sort of what
you'll be innovating on at that point.
1:17:08 You've mentioned that every four
years of like, you're, you're staring
1:17:11 at a problem where you feel like,
ah, there has to be a better way.
1:17:15 So can't wait to see which sort of absurd
things you might build in the future.
1:17:20 So you mentioned that right now
you're at Stripe and I think at Stripe
1:17:24 you're working on design systems
and like more UI related things.
1:17:30 So which sort of things are
you, are you working at Stripe?
1:17:33 Yes.
1:17:34 So I work on our design system.
1:17:36 It's called SAIL.
1:17:37 And so SAIL recently, it used to just be
a single team called the design system
1:17:41 team, but now there's been changed into
like a larger org, which is really cool.
1:17:45 Exciting in some ways, it kind of, you
know, messed up our team a little bit.
1:17:49 Some, some people got moved
around, which was, we had to kind
1:17:52 of work through that, but we did.
1:17:53 And like, there's a wholesale
org now, which is great.
1:17:56 So it's like a, a bigger investment.
1:17:58 SAIL itself is becoming this thing.
1:18:00 That's actually more than
just a UI design system.
1:18:02 We're also starting to integrate other.
1:18:04 Concerns that you always hit when you
start doing front-end work, which is like,
1:18:08 how do you internationalize something?
1:18:09 How do you, how do you do observability?
1:18:11 How do you do like GraphQL queries or how
do you just fetch data like in general?
1:18:15 And so SAIL is becoming like
a bigger platform overall.
1:18:18 I am focusing more on the, on the
UI design system part, but I'm
1:18:21 also collaborating a lot with the
others as well to sort of integrate
1:18:24 this into a cohesive platform.
1:18:26 Internally, we have a, a.
1:18:27 A whole system for like
variants and tokens.
1:18:30 And we have a, like a
view and a, and a CSS API.
1:18:33 And it's something that I don't
think we've talked a ton about.
1:18:36 I think we really should talk
about it more, um, openly.
1:18:38 And cause it's, it's, it's pretty neat.
1:18:40 We try to.
1:18:42 Use third party libraries when we can.
1:18:44 We actually leverage a lot of React Aria.
1:18:46 So a lot of our components
use, use React Aria.
1:18:48 Yeah.
1:18:48 We actually use the, also the like
lower level hooks API, which is like in
1:18:51 some ways really cumbersome to use, but
like, that's kind of intentional, like,
1:18:55 because it gives you like direct access
to the entire way that the things work.
1:18:59 They now have a higher level API, like
the React Aria components API, which
1:19:02 is all like really, really great.
1:19:03 But we do things with React Aria
that I don't think anybody else does.
1:19:07 I think that like, there's like
classes that we import and use.
1:19:11 That, like, we've talked to the
ReactARIA team and they're like,
1:19:13 nobody else has imported that class
and, like, used that directly.
1:19:16 There's, like, a list, like, a list
collection class that is, that it uses.
1:19:19 And we import that and create
our own collection system.
1:19:22 And, like, cause, like, Stripe
has pretty specific needs, right?
1:19:25 And so what I love about ReactARIA
is that it's taken this, like,
1:19:28 unabashed approach to being cumbersome.
1:19:31 And it's, like, kind of intentional.
1:19:33 Like, if you look at the docs for some
of the hooks, like, it's a lot of code
1:19:36 to get a, Menu working, but because it
is that openly exposed, you have like
1:19:43 total control over how everything works.
1:19:45 So it's really allowed us to go
in and really, really wire things
1:19:47 up the way that we want to.
1:19:49 So yeah, it's, it's great.
1:19:49 It's a fantastic library.
1:19:51 It's super, super good.
1:19:52 Um, lots of good accessibility
that, that you get from that.
1:19:55 But that's our component system, right?
1:19:57 We still have a lower level,
like view and CSS APIs.
1:20:01 And so how you do CSS, how you do tokens,
how you, how you do variants, how do
1:20:04 you make sure that you, when you, uh,
create a new component, that everything
1:20:09 is wired through such that like, The, um,
like if you pass styles to it, but you
1:20:15 also internally want to style the, the
same div, like, like the same top level
1:20:19 div that you take the styles from your
props and you apply styles to that div.
1:20:23 But like in that component,
you also want to apply styles.
1:20:27 Do that div.
1:20:27 How do you, how do you
combine those styles?
1:20:29 I guess you could like spread on the
styles that you get from the props, right?
1:20:33 And then spread on your
own styles as well.
1:20:35 But then you get into like
precedence problems, like which
1:20:38 order do you spread those on?
1:20:40 So we have a whole way of like creating
a component, a whole pattern for
1:20:44 taking the props that you get and.
1:20:46 And spreading, and like, spreading
those props onto some internal element.
1:20:50 And we have a very strict precedence
order for how styles and how
1:20:54 variants and, and how other things
like that get merged together, um,
1:20:58 in a way that's a very intuitive.
1:21:00 Um, and it's all very, very consistent.
1:21:01 So we have like a, an
API called create view.
1:21:04 So that's how you create a new component
that interacts with our whole system.
1:21:07 When you call create view, you get back
an exact same React component, just like
1:21:11 you would call it forward ref, right?
1:21:12 Like it's, it's taking a component, but
it's giving it additional capabilities.
1:21:16 And the, the, the two things, like
there's a couple of things that
1:21:18 it gets, it gets like a CSS prop.
1:21:20 So we, we pass CSS by, by saying
button CSS equals, and then an object.
1:21:26 And in that object, we have a
whole like little system that
1:21:29 is wired up with our tokens.
1:21:30 And so you can say margin
is, you know, small.
1:21:34 And small gets resolved to a token.
1:21:35 We also do like things where you
can, um, say like small is eight
1:21:39 pixels and we have an ESLint rule
that automatically knows that for
1:21:43 the current theme and for the current
system wired up, small is eight pixels.
1:21:48 So you've hard coded a token here and it
automatically actually fixes that for you.
1:21:52 So it's changes eight pixels to small.
1:21:55 So we, we try to let developers build.
1:21:58 Like not get in their way.
1:21:59 Basically, we really want them to
sort of, we have the same, like fall
1:22:02 into the pit of success where you
should go off and build a product
1:22:05 the way that you want to build it.
1:22:07 And sale should meet you where you
are there and like, let you build
1:22:11 components that end up getting wired
up the way that we want them wired up.
1:22:15 But just because that's the
The way that feels natural.
1:22:18 Like you shouldn't have to feel like
you have to like go against our system
1:22:22 and like begrudgingly use it to like
be conformed to Stripe's design system.
1:22:26 Right.
1:22:26 We want you to be like happy using it.
1:22:28 And so a lot of our work goes into
like making the APIs feel good.
1:22:32 Yeah.
1:22:32 There, there's a lot of stuff here
that I think we should talk about more.
1:22:34 Like we have our own variant system
and our whole, like how tokens are
1:22:37 wired up is really, really interesting.
1:22:39 Cause one thing about Stripe.
1:22:41 It's a very, very broad company.
1:22:43 We have a lot of different products.
1:22:45 We have the dashboard.
1:22:46 We have connect.
1:22:47 We have like people taking
pieces of functionality and
1:22:50 embedding it in their own website.
1:22:52 And then they're theming that
content, but it's like exactly the
1:22:54 same little widget that you would
take from the dashboard, right?
1:22:57 It's like a payments list that
you would take from the dashboard
1:22:59 and embed into your own page.
1:23:01 And then we also have like custom hosted.
1:23:04 Invoices and a bunch of other
little like third party things.
1:23:06 Then like checkout and like elements
as a whole, another thing as well.
1:23:09 We're trying to bring all of that
together into us and to use the same
1:23:12 design system that can be themed and like
leverage and customized for each surface.
1:23:17 And so it's really hard problems.
1:23:20 Like honestly, and it's, it's, it's
a lot of work, but it's really fun.
1:23:22 That, that sounds fascinating.
1:23:24 Uh, I feel like I want to learn a lot
more about how to do design systems.
1:23:29 I should also educate myself more of
like, should I use a design system
1:23:33 even for a smaller scale products?
1:23:35 For example, would you now
looking back, would you've used
1:23:37 a design system for Actual?
1:23:39 Is this like, rather like, does it
solve an organizational problem or does
1:23:43 it really help even on a smaller scale
when fewer engineers are involved?
1:23:48 But I most certainly love the design
principle of like the pit of success
1:23:52 that can't go wrong with that.
1:23:54 Yeah, it's great.
1:23:55 It's, it's been a good, like a
good way to frame things for sure.
1:23:58 So you're building with React,
uh, Stripe it seems, and you've
1:24:03 also used React for Actual.
1:24:05 React has changed quite a bit over
the years, so I'm curious to hear
1:24:09 whether you have any opinions on
where React has gone over the last
1:24:15 Opinions on React
1:24:15 I don't have super strong opinions
only because I feel like I
1:24:18 can't back them up right now.
1:24:20 I haven't given it the time to
sit down and really like write
1:24:23 out like a thorough argument for
why I should feel a certain way.
1:24:27 I think it's been hard for just
in general, like everybody.
1:24:29 I respect the team in a
lot of ways because I think
1:24:32 they have a high bar, right?
1:24:34 Unlike many other teams.
1:24:35 Developers, generally speaking,
I feel like we don't have a high
1:24:38 enough bar and I feel like the
React team has a high bar and like,
1:24:41 ultimately, I do respect them for that.
1:24:43 I do think there's, there's been
times when it's just like very heavy
1:24:47 investment and very complicated
things when it feels like there's
1:24:50 lower hanging fruit, which are like.
1:24:53 Man, this really sucks to have to do this.
1:24:54 Like every single time sucks to have
to use forward ref every single time.
1:24:58 Why can't just ref be a freaking prop?
1:24:59 And like, I know that there's like
backwards compatibility problems and
1:25:02 like all sorts of reasons, but it, when
like two years are spent on this really
1:25:05 complicated thing, and like, we're still
hitting these problems on a day to day
1:25:09 development thing, sometimes there's a
little dissonance there that can be a
1:25:13 little bit like, ah, you know, we also
don't pay for react, it's almost like.
1:25:17 I don't know, like we're
getting it for free.
1:25:18 It's up to them to ultimately decide.
1:25:20 We're not, they're not
forcing us to use React.
1:25:22 There's a little bit
of a lock in for sure.
1:25:24 Like I don't know how Stripe would
possibly move away from React, but
1:25:27 ultimately like the members of the
team, I respect really, really well.
1:25:30 And I'm not going to say
anything like super bad about it.
1:25:33 I think react server components
is really interesting.
1:25:35 Again, it's like, sure, like the
local-first stuff might be able to
1:25:38 be cleaned up a little bit with it.
1:25:39 Like, there's kind of some interesting
things there where like, maybe you
1:25:42 could like run components on the server.
1:25:44 Like, even if it's just like a
backend web worker process, but
1:25:47 the wins are much less for sure.
1:25:49 Like.
1:25:50 Everything's already local.
1:25:51 I don't care if it goes
through a WebSocket message.
1:25:53 Like I can, like, you
can embed SQL queries.
1:25:55 Like we're embedding SQL queries already.
1:25:57 We don't need React
Server components for it.
1:25:58 So it's less convincing to
me if you're already doing it
1:26:01 the way that we're doing it.
1:26:02 But for Stripe, like React
Server components could be
1:26:04 potentially pretty compelling.
1:26:06 But it's still, again, overall,
like, Man, I just want support
1:26:10 for exit animations, right?
1:26:11 Like I want the ability to not have
to wrap something in animate presence
1:26:15 to just freaking get something to
like maintain it in a dom while it
1:26:19 animates out and then unmount it.
1:26:21 Like React still doesn't
allow you to do that.
1:26:23 Like exit animations are
a huge pain in the butt.
1:26:25 Yeah,
1:26:25 I, I fully agree.
1:26:26 And I think the React server components,
what you've mentioned, I think the,
1:26:30 the way from my current perspective,
how it would most meaningfully help
1:26:34 in a local-first context as well.
1:26:36 is on the initial upload experience,
since that is sometimes a bit of
1:26:41 like the cost that you're paying that
you're, you're kind of with local-first,
1:26:45 you say like no more spinners.
1:26:47 Well, sort of, since you have like one
front loaded, typically larger spinner.
1:26:52 And I think this can be addressed also
with react server components, where
1:26:55 you get sort of an hybrid approach.
1:26:57 Where you get much more quickly reactive
initial version of the app that then
1:27:03 upgrades itself to be local-first.
1:27:05 So that, that's my take on react
server components, but the other pain
1:27:09 point that you've pointed at with
like exit animations, et cetera, and
1:27:14 where this is kind of where we are
now paying the cost for virtual DOM,
1:27:18 which constantly updates everything.
1:27:20 I've had actually some, some
interesting results now with just
1:27:24 using web components for that.
1:27:26 Where it's a bit funny since like, in
some regards, it feels like going a bit
1:27:30 back in history where your DOM is more
static in a way, but this way you actually
1:27:36 also can think a little bit differently
about like, for example, animations.
1:27:41 So, and this is where I feel also on
the general theme of learning more from
1:27:47 other native programming environments,
other native platforms, where for
1:27:52 example, on iOS, when you have like.
1:27:55 Things like a collection view or
a table view controller where you
1:27:58 have those cells and they're like,
they don't constantly cycle out.
1:28:03 They cycle out if you're like, if they
leave your view, but based on like
1:28:07 entering the view, leaving the view,
this is how we can do animations.
1:28:10 And with an approach like web
components, you can actually do that by.
1:28:15 Using the native aspects of the web, like
native to the platform web, much more.
1:28:20 And I feel we're now fighting
a bit of an uphill battle to
1:28:23 get those benefits from React.
1:28:25 Agreed.
1:28:26 Yeah.
1:28:26 It's interesting.
1:28:27 There's a decent escape
hatch there, right?
1:28:29 Where you can mount a div and then in
an effect, like you can do whatever you
1:28:33 want there, which is, which is nice,
especially for a company like Stripe,
1:28:35 because we can sort of like opt out when.
1:28:38 Needed, but something does like, there are
certain things that are like systematic.
1:28:42 And, you know, as something, somebody
who works, works on a design system,
1:28:44 seeing like animations are not something
that you can just like opt out for.
1:28:48 Right.
1:28:48 That's something that you apply on
like almost every single, I don't
1:28:51 know, like a 10th of the elements
on the page, which is a lot.
1:28:55 have animations.
1:28:56 You can't just like opt
out when you want to.
1:28:58 So it's, it's hard.
1:29:00 Yeah.
1:29:00 I would love to see
some improvements there.
1:29:02 Just recently on my blog, I have quick
and dirty type thing, but like I don't
1:29:05 use re I use react, I use, um, Remix.
1:29:08 And so I use reactor on the backend,
but I don't use it on the front.
1:29:10 It just feels so fast and nice.
1:29:11 It's just a.
1:29:12 Just don't do it just content, but like I
am starting to add more interactive parts.
1:29:16 And so I'm sort of playing around with
like, well, okay, what do I do here?
1:29:20 Do I need to load in the full react?
1:29:22 Maybe this is where like partial
hydration could be interesting.
1:29:25 Uh, but like, I just don't, it's my blog.
1:29:26 I don't need partial hydration on
my frigging blog, but I do want
1:29:30 to be able to like, um, there's
like, I have like live interactive
1:29:33 demos and I want to be able to.
1:29:35 Have a view source button.
1:29:36 And when you click that, it opens up
this, like, it basically like zooms
1:29:40 into the interactive demo to where it
shows the demo, like, like the demo is
1:29:44 still running, but it's running in a
dialogue and the demo is run on the left.
1:29:48 And then on the right is the code
for the demo and then all sorts
1:29:51 of other controls for the demo.
1:29:53 And the way that I implemented that
was I literally transport that real
1:29:57 Dom node, I take the Dom node instance
itself from the, like the whole
1:30:01 demo Dom node, and I, I move it.
1:30:04 I create the dialogue and I put
the code on the right, and I moved
1:30:08 the DOM node into the dialogue.
1:30:10 And the demo just keeps
on running just fine.
1:30:12 Like it, it moves into the dialogue.
1:30:14 And I did that in like 10 lines of code.
1:30:16 And there is a certain amount of
like, Oh man, it feels so good to be,
1:30:20 yeah, I just have access to like this.
1:30:22 Dangerous, dangerous API APIs, but like
I, to do this in react would have felt
1:30:26 really backwards, like bending backwards
and doing all sorts of weird things.
1:30:30 And then you can like, and you
close it and it moves it back and
1:30:32 it just keeps on running just fine.
1:30:33 Right.
1:30:33 And I, I don't know, to, to, to think
about it in the react mental model,
1:30:36 I would have been like, Oh, okay.
1:30:37 I need to create like a portal,
create a bunch of components and
1:30:40 like wire them all, I don't like
just saying like dom node, remove
1:30:44 node, and then like demo dialogue.
1:30:46 append.
1:30:47 Uh, I completely agree.
1:30:48 I feel like we've been now over the last.
1:30:51 React has been now around for 10 years.
1:30:53 I feel like over the last 10 years,
we've been really leaning to use the
1:30:59 React hammer for like every little thing.
1:31:01 And I think that brought a whole lot
of benefits to us, but I think we
1:31:06 don't question it really anymore.
1:31:08 And like, we, I think this is
like the only way to go about it.
1:31:11 And the web has since then really evolved.
1:31:14 Like we've gotten a lot more,
the primitives in the web
1:31:18 itself have gotten a lot better.
1:31:20 And so I think web components
have also come quite a long way.
1:31:25 They haven't seen as much investment
in terms of building a nice ecosystem
1:31:29 around it, but I think it's the closest
we got to a native feel in the web.
1:31:35 And, uh, I think sometimes you'd be
surprised how simple things can be if you
1:31:40 embrace those primitives more directly.
1:31:42 One little anecdote I want to share
there, a friend of mine, Cheng Lu,
1:31:45 who's I think also worked on the React
team for a while, he did a couple of
1:31:50 really fascinating demos and I think he
launched some of them on, on Hacker News.
1:31:55 Where he, for example, built a photo
gallery and a Hacker News clone as well.
1:32:00 And so when you try those apps, those are
the smoothest animations and the smoothest
1:32:05 feel you've ever seen on the web.
1:32:07 And you kind of feel like,
what is going on here?
1:32:10 Is this like rendered to web
GPU using Wasm or something?
1:32:14 And turns out this is all
just normal DOM, normal CSS.
1:32:18 He's, he's built this like
very simple, elegant system.
1:32:21 I think the photo gallery is like in
a hundred or 200 lines of JavaScript,
1:32:26 like no dependencies, nothing.
1:32:28 And this feels like a total native app.
1:32:30 So this really reminded me of like
how capable the web really is.
1:32:34 If we use it directly without
layering too many things in between.
1:32:39 I have high hopes for, for the web.
1:32:41 If we set up, set ourselves a high bar.
1:32:43 Same.
1:32:43 Yeah.
1:32:44 And I think React needs to figure out
how to like move with the web without
1:32:48 breaking backwards compatibility,
find the right trade off there.
1:32:51 Like, I guess even the like web
component support type stuff in that
1:32:54 has been like super long in the making.
1:32:56 So I think that's something that.
1:32:58 Outro
1:32:58 hey, James, you've been very generous
with your time and walking us
1:33:02 through like your entire journey of
the, the various chapters of Actual
1:33:06 from Electra and the mobile apps,
the web, absurd sequel, et cetera.
1:33:11 So thank you so much for your time.
1:33:14 If there's anything you want to share
with the audience here, any sort
1:33:17 of shout out, now's a good time.
1:33:19 Uh, cool.
1:33:19 Yeah.
1:33:19 Thank you for having me on.
1:33:20 This was amazing.
1:33:21 I mean, honestly, shout out to
Martin Kleppmann, PVH, Peter.
1:33:25 I found them through
their local-first essay.
1:33:28 And I, I came and give like a, gave like
a workshop at their research studio and
1:33:33 like talk to them a bunch since then.
1:33:34 And they've always been very encouraging
throughout the whole process.
1:33:36 So super fun to talk to
them about all this stuff.
1:33:38 Awesome.
1:33:39 Thank you so much for your time
and coming on the show today.
1:33:42 Thank you so much.
1:33:43 Thank you for listening to
the localfirst.fm podcast.
1:33:46 If you've enjoyed this episode and haven't
done so already, please subscribe and
1:33:50 leave a review wherever you're listening.
1:33:52 Please also tell your friends about it.
1:33:53 If you think they could be interested
in local-first, if you have feedback,
1:33:57 questions or ideas for the podcast,
please get in touch via hello at
1:34:00 localfirst.fm or use the feedback form on
our website, special thanks to Expo and
1:34:06 Crab Nebula for supporting this podcast.
1:34:09 See you next time.