localfirst.fm

A podcast about local-first software development

Listen

Conference

#5 – Kyle Mathews: Benefits of using a sync engine, personal local-first apps, ElectricSQL

The guest of this episode is Kyle Matthews, who, in the past, founded Gatsby JS and is currently delving into local-first software. 
In the conversation, Kyle shares his experiences in building some small-scale local-first apps for his personal use and discusses how the utilization of a data syncing engine liberates significant development time.

Mentioned in podcast


Links:

Thank you to Expo and CrabNebula for supporting the podcast.

Transcript

Introduction
00:00And so you're like, Hey, API, give me the data from my widgets.
00:04And then I render it.
00:05So it's sort of an imperative step, kind of like manually telling the client,
00:10tells the server, give me that thing.
00:12Where sync engines, it's a a move from an imperative to declarative.
00:17So instead of saying, give me that data, it's sort of like for this page,
00:22I declare, I need this data and then the sync engine makes it so, often
00:25run into the problem where like, just like sort of getting data is easy,
00:28but subscribing to updates is hard.
00:30Welcome to the local-first FM podcast.
00:33I'm your host, Johannes Schickling, and I'm a web developer, a
00:36startup founder, and love the craft of software engineering.
00:39For the past few years, I've been on a journey to build a modern, high quality
00:43music app using web technologies.
00:45And in doing so, I've been falling down the rabbit hole of local-first Software.
00:50This podcast is your invitation to join me on that journey.
00:54In today's episode, I'm speaking to Kyle Matthews, who's in the past
00:58has been the founder of Gatsby.
01:00js and is now exploring local-first software.
01:03In our conversation, Kyle shares his experience building some small scale
01:08local-first apps for his personal use and how using a data syncing engine
01:12frees up a lot of his development time.
01:15Before getting started, also a big thank you to Expo and Crabnebula
01:19for supporting this podcast.
01:21And now my interview with Kyle.
01:24Hey, Kyle, welcome to the show.
01:25How are you doing?
01:26Uh, doing great.
01:27Uh, glad to be here.
01:29Thank you so much for, for coming on.
01:31You're another fellow early Local-Firster.
01:34So I'm very curious to hear about your background and then hear
01:37about what led you to Local-First.
01:39So, uh, I've been doing web development of various sorts
01:44for like the last 15 plus years.
01:46I've seen a lot along that way, you know, like the whole jQuery era, did Backbone.
01:50js for those who read history books and, you know, it was early React, GraphQL.
01:55I actually started with Drupal.
01:57Funny fact, Drupal, Drupal was big back in 2000, still widely used, but
02:00people are like a little more expansive about their view of Drupal back then.
02:04Then switched to kind of JavaScript, single page apps,
02:06like really early 2010 ish.
02:08And, uh, yeah, I've been kind of writing that ever since.
02:10Yeah, then, then of course, uh, Gatsby 2015 introduced that, uh, which kind
02:15of like, Took react and it was sort of the first proper production ready,
02:19you know, react, uh, meta framework.
02:21And so kind of rode the whole react, uh, wave for a long time.
02:25But yeah, in general, my interest is what are the tools we use to build software?
02:32Cause like, I think software is very important and.
02:34The quality of the tools that we use determine the quality
02:37of the software that results.
02:39We shape our abilities and our ability to shape us.
02:41It's sort of like we, we, we shape our tools and then the
02:44tools shape everything else.
02:45Cause I think there's really like high leverage to kind of
02:48continually pondering that question.
02:49Cause if we can genuinely move to a genuinely better tool, then
02:55there's enormous Effects that result from that, not just in our actual
02:59day to day life, but like, you know, we're a software creators.
03:02Like there's a lot, a lot of people whose lives are affected by software.
03:05So yeah, I've seen like genuinely better tools emerged in sort
03:08of my, my software career.
03:09So like, I think react was a massive step forward.
03:12I think Grackula was a smaller step forward, but also a genuine step forward.
03:15And I feel like, you know, local-first, it feels like this is
03:19another sort of massive step forward.
03:20Leap forward, um, and how we build software.
03:23And I think it's something that in the coming years, more and more software
03:27will be kind of built in this fashion.
03:29I definitely agree.
03:30I mean, uh, I'm also definitely a connoisseur of good tools, but luckily
03:36there's a lot of good tools and many different areas of development overall.
03:41Like you just.
03:42You can also look at other language ecosystems.
03:44I'm definitely very interested and inspired by all the good stuff that's
03:49happening in the Rust ecosystem, but also super cool ideas that you find in
03:53other ecosystems, such as like the Elixir ecosystem with live views, et cetera.
03:58So there's like a lot of great tools and great ideas, uh, in all of those places.
04:03But I'm curious, which sort of problems have you experienced
04:08in the past that has led you.
04:11To, to local-first in particular, and where you now see a different approach,
The State Transfer Problem
04:17You could sort of think of like client server development,
04:20there's sort of three problems.
04:21There's sort of the UI, like how do we construct the UI?
04:25How do we do interactivity, charting stuff?
04:28So there's a lot of like different areas within like the UI and then
04:31there's sort of backend problems.
04:32There's a lot of like complicated computing bits.
04:34There's like, how do you run infrastructure at scale?
04:38How do you.
04:39Make things efficient.
04:40So you're not like, you know, spending enormous amounts of money, uh, security.
04:44There's endless domains of problems in the backend.
04:47Um, and then there's kind of like the middle bits, which is like state transfer.
04:51Like, how do you move bytes back and forth?
04:54Cause generally the client can't hold all the data.
04:57So you have to like, say, I need part of the data to be pulled over.
05:00And like, generally things are changing to like, how do you get that data from
05:03the server to the client and so on?
05:06And so there's a lot of.
05:07Problems around moving state back and forth.
05:10And so, yeah, so I think I'll go first.
05:11It's interesting because I feel like the front end and the backend, there's been
05:16just enormous number of innovations, both sides, all the investment and,
05:21and, and react it's felt and solid and view, uh, and like VEET and Webpack.
05:26And there's just like enormous investment that's gone into making the front
05:30end experience dramatically better.
05:32Plus all the, you know, all the libraries, there's like NPM has like Millions
05:35of libraries or something like that.
05:36I don't know.
05:36Like each of those things is just like enormous effort from individuals
05:40and groups all across the world.
05:42And it's made the UI world dramatically better place.
05:46Like it's, it's just remarkable to me how much faster it is to build.
05:51Amazing stuff versus like 15 years ago.
05:54Like I, the first like big JavaScript dashboard I built, it took me like
05:58six months to ship the first version, partly because I wasn't a very good
06:01programmer, but partly because like, there was just like no guidance anywhere.
06:05Now there's just like endless starters, endless use cases.
06:08Like React is such a compact, simple abstraction.
06:11It's dramatically better.
06:12And that's because there's been like tons of innovation there.
06:15And the backend too, there's like all the cloud giants are pouring tons of research
06:18into stuff and there's like tons of.
06:21Individual cloud services where before you're like, Hey, that's
06:24three months of programming.
06:24And now it's like two hours to implement an API and off you go.
06:28So there's been lots of innovations too in the back end, but yeah, the state
06:31transfer bit, like the, you know, moving bits back and forth hasn't really, I
06:36don't know, got as much investment.
06:38Partly.
06:38I think it's like, it's not always a problem.
06:40Often.
06:40It's pretty simple.
06:42You know, if you're just sort of like, Building a dashboard.
06:44You're like, get me the data for the dashboard, render the data.
06:48It's not a very complicated problem oftentimes, but I think what we're seeing
06:52is that we're like the best teams, the best apps investing disproportionately
06:58their time compared to everyone else, because that's kind of where, where the
07:01best teams see a kind of an advantage that they can like build something
07:04significantly better than other people.
07:06And so what I think you're seeing now is like the best.
07:10Apps in the industry, you know, like the Figmas, the Linear, Superhumans,
07:13et cetera, they're pointing now towards sort of state transfer, like, how
07:19do you load your app extremely fast?
07:21Because everything's cached locally.
07:23How do you have real time sync between, you know, different people?
07:26So you can build like, rich collaborative experiences.
07:28How do you, you know, save locally and then sync backwards?
07:32So you're not like, Doing little spinners all the time when you save stuff.
07:35So it's like always fluid and fast.
07:37And they're all kind of arriving at the same idea, which is like, Hey, this
07:41whole like rest, you know, like posting things back and forth, pushing bites
07:46around manually, doesn't cut it anymore to build like really complicated, amazing
07:51apps, you need a higher abstraction.
07:54You need something more sophisticated for.
07:57Handling that state transfer and, and that's sort of like this local-first and,
08:00or, you know, like sync engines, it's kind of like the component that sort of
08:03handles that, that state transfer back and forth between the client and server.
What are Sync Engines?
08:08That's super interesting.
08:09So you, you mentioned the term sync engine that, that might be not
08:12familiar with everyone in the audience.
08:15So could you elaborate a little bit what a sync engine is
08:18and how that might relate to.
08:21The, the approaches that you've mentioned before.
08:24I feel like it's the best term for uniting a bunch of different
08:29tools and their approaches.
08:30But yeah, the basic idea is that REST API, Grackle API, TRPC, whatever.
08:37They're all based around rendering a page for widgets.
08:40It's a widget page.
08:42And so you're like, Hey, API, give me the data from my widgets.
08:46And then I render it.
08:47So it's sort of an imperative step, kind of like manually telling the client,
08:52tells the server, give me that thing.
08:54Where sync engines, it's a a move from an imperative to declarative.
08:59So instead of saying, give me that data, it's sort of like for this page,
09:04I declare, I need this data and then the sync engine makes it so, and often
09:08run into the problem where like, just like sort of getting data is easy,
09:11but subscribing to updates is hard.
09:13Like everyone kind of notices there's a massive gulf and
09:16difficulty between those two things.
09:18And so what's cool about sync engines is that because it's sort of more of a
09:23declarative step, there's actually no additional complexity to get subscription.
09:27This guy is the same thing.
09:28You're just saying, I need this data.
09:30And then the data shows up.
09:31It's a new abstraction for how data.
09:35Arise on the client where, yeah, again, you're going from imperative to just
09:39being like, you're describing the shape of data that you need to show
09:42up on your site for a particular page.
09:45I know another good analogy.
09:46It's very similar to shift from like jQuery to react where jQuery, you're like,
09:51get this DOM element and tweak the text.
09:55Or get this DOM element, remove the child, construct two other children, drop it in.
10:00You're like a little kid playing with like blocks or something.
10:03And you're like, you're like fiddling all the time, which was actually fun.
10:06I actually enjoyed it.
10:07It was a fun little challenge, but it's a huge waste of time where, you know, we're
10:10moved to React and React is more like, I don't know, I want it to look like this
10:13and just sort of happens somehow, like React does a bunch of weird magic under
10:17the hood and it just kind of efficiently somehow shifts everything around.
10:21Uh, without tearing and voila, it's there.
10:23You're like, cool, it's there.
10:24And yeah, that's just a lot less to think about.
10:26When you're just like, I want something to be so, and I don't really
10:28care about the complicated details.
10:30That's a lot less to think about.
10:32Which is the value of abstraction?
10:33Because we, we have a finite amount of things we can think about.
10:35So, if we drop two, then we can like, replace it with, Two other stuff instead.
10:40So if you drop a lot of thinking about state transfer, then you have more
10:43time to think about other things.
10:44So you've now explained a little bit, the benefits of a sync engine and
10:50that it simplifies a lot of things.
10:51But for someone who's now like curious, how does this actually work?
10:56Can you explain what a sync engine does and how it does it?
11:01I'll, I'll use an example for the one that I've used the most electric SQL.
11:04So electric SQL is sort of Postgres based and you have, you know, Your tables
11:09and basically the shape is like kind of declared like this is the shape of data.
11:13I want polar.
11:14It's just like a sequel query.
11:15So you're just like, I want.
11:16this table and rows that match this criteria.
11:20And then the sync engine kind of like takes that and then it says like, okay,
11:24I will send you sort of the initial bunch and then any updates that I see from then
11:28on that match that shape off it goes.
11:30That makes sense.
11:31So, and this would be one particular example, one particular technology
11:35for when your starting point is an existing Postgres database.
11:40So is that the way how I should think about it?
11:42That like I have my And like traditional way of how I'm dealing with data in
11:48a more global state, for example, Postgres, and then I'll just slap
11:53a sync engine on top of it, connect my clients to it and off I go.
11:57In theory, in practice, all these tools are like, this
12:01whole space is pretty new still.
12:03So I think that's sort of the platonic ideal that everyone's working towards
12:06is like, you just sort of slap a sync engine on and off you go.
12:09And that does work.
12:10In some parts, you know, all these tools are like, there's some limitations
12:14that you have to kind of live with and that they're working to, to lift, but
12:17there's understandably a decent bit of complexity to kind of like take this
12:2130 year old, whatever old Postgres is technology and toss a new thing onto it.
12:26As long as you live within the abstraction, I kind of, or live within
12:28sort of the existing limitations, then yes, that is actually the very,
12:31very most of the reality I've been building, like a number of sort of
12:33toy stuff, both to play with this local-first approach, Sinkage approach.
12:37But also, you know, exploring a number of other things and it is like very simple.
12:41You just say like new table and then it creates a client side library.
12:47Kind of like very Prisma esque.
12:48Uh, actually they're still using, I think, Prisma under
12:50the hood to generate the client.
12:52And then you just immediately just start reading and writing
12:54from like the local SQLite.
12:56So because they use Postgres in the back end and then SQLite.
13:00It's very straightforward.
13:01Just like write a query and data's there, do an insert off it goes.
13:05So I can't promise it'll work for like, yeah, anything will just work.
13:08But if you're living within sort of the current restriction, it is
Personal Apps - Pt 1
13:13That sounds great.
13:14And I think there is a lot of different approaches, how a sync system could work.
13:19We've also learned a bit about auto merge in previous episodes, but I'm curious to
13:23hear more about your experiences building apps with the stack that you've mentioned,
13:28can you tell a bit more about the apps that you've tried those new approaches?
13:33Uh, yeah.
13:34I mean, none of these apps have been, we've all been for like
13:36personal use, so they're not like anything earth shattering.
13:40The most kind of.
13:42Complicated one is a one I'm just actually finishing up and my wife had been
13:46working on it and it's like a kitchen AI.
13:49So basically you, you can like take a photo of different ingredients in
13:54your kitchen and then the AI will then sort of like convert those into
13:57ingredients in the store and then you can set like, what is the fill level?
14:01When's the expiration date?
14:03And then when you want to go do, do a recipe, you can upload, like
14:07you kind of copy and paste the recipe into it and it kind of like
14:10parses out all of the ingredients.
14:12And then what's interesting is like, it's actually kind of like ingredient
14:15or recipes all kind of describe ingredients slightly differently.
14:17So like just doing like a naive matching or something.
14:20You won't actually work, but then, so I'm using like embeddings to, to
14:24kind of do a more fuzzy semantic match between the ingredients in the kitchen
14:28and like the recipe ingredients.
14:29And so then I'll say like, Hey, here's all the ingredients you have.
14:32Here's the ones you need to buy, but also sort of like tell you, it's
14:35like, Oh, Hey, you're running low.
14:36You should double check if you have enough of this one, or, Oh, this
14:38one is like expired, so you should probably just get some new spice.
14:41So you're, you're, cause the old one doesn't taste very good anymore.
14:45Anyways.
14:45And then it's sort of a one click.
14:47You know, create a shopping list with, uh, we're using Trello.
14:50So just sort of like sends it off to Trello.
14:52Another cool thing that we did is like, we just give it an enum
14:55of sections in the grocery store.
14:57So it's like, there's meat and seafood and dairy and produce
15:01and canned goods and stuff.
15:02And so then each ingredient is coded to one of those sections.
15:07And so then when you go to the store, you have this like neat list of.
15:11Uh, the different things, so it's actually pretty fun to use.
15:14Uh, it'd be pretty fun to build, but, but yeah, so, so the actual
15:17process of building it is, you know, the local-first stuff, the electric
15:20SQL stuff is defining my schema.
15:24Well, another cool thing about all this too, is it's very data centric.
15:26I've noticed a big shift in how I build stuff now versus before is it
15:31used to be steady at the back end and like the database, everything was like
15:35annoyingly complicated that I just sort of delayed it until I was sort
15:40of more or less finished prototyping.
15:41So what I would do is like, just keep all the data in memory in the
15:45browser and then build the UI, try a bunch of stuff and just kind of like.
15:48Have a little JavaScript rays and objects, you know, whatever.
15:51And then finally I was like, okay, now I kind of like know what my
15:53data structure is and everything.
15:55And now I'll just actually set up the database and start
15:57persisting it and whatever.
15:59But now it's like so easy with electric SQL to kind of.
16:04Set up the database and have a client and have everything syncing that like,
16:08like the very first thing I do is write my crate, you know, table stuff and,
16:12and, and I have the tables and which is a huge improvement because it's very
16:15nice having a fully typed database, you know, I, because you have all the
16:20Postgres stuff you can like, you know, pull up PSQL and start writing queries and
16:24anyways, there's a lot of power you get.
16:26For free and you have like type checking, you have, you
16:29know, foreign key constraints.
16:31So you have kind of all these nice things that database give you from
16:33day one while you're just prototyping.
16:36Normally what I do when I'm prototyping is I just blowing
16:38away the database and restarting.
16:39But that again is like 30 seconds or something, um, to do.
16:43It's also quite fun to like chat GPT is like great at like, Creating SQL.
16:48So I use that a ton when I prototype and I just sort of like, just kind of
16:51freehand describe all the data I need, like, and then it spits out, you know,
16:55like five create table statements and I just like pop it in and off I go.
16:59It's very, very productive.
17:00Uh, and it's quite fun just to have a full blown typed database
17:05backed, you know, SQL first reactive system while you're prototyping.
17:10Yeah.
17:10It's both like a very productive prototyping system, but what's
17:12genius about, of course, about the whole thing is like your prototype.
17:15That is immediately your production system too.
17:17Cause I can just like, as soon as I'm done, like, I don't have to like,
17:20fix up a bunch of data problems cause it's, it's been good from the get go.
17:23I just ship it and, and it works perfectly.
17:26And I'm not like, Oh no, I have all this like crap data now that's
17:29accumulated production system.
17:30I took all these shortcuts, you know, to like work on my prototype.
17:32It's just like, it's just, you have all the database
17:34guarantees from, from the get go.
17:36I, I love that story.
17:38I think this is so powerful and the analogy you've provided before
17:43in terms of going from jQuery where you have to do everything manually
17:46to React freeing you up to no longer having to do all of this manually.
17:51Now that really makes sense to me because now you could like you've, you've
17:56mentioned a few of like the, the new, like AI aspects of this, or that the app
18:01is an app just for you and your wife.
18:03And that's totally fine.
18:05Like you're basically now freed up by so much data wrangling of before you
18:10would need to like maybe set up a GraphQL endpoint, a REST endpoint, and you'd
18:14spend so much time just to have all the, the data moving stuff taken care of.
18:20And if that now just works in the same way as React.
18:25Automatically working and keeping your views up to date.
18:29And now if we elevate to the next level of Muslow's hierarchy of, of needs
18:35for, for app development, and we have the data automatically taken care of
18:39now, you can focus actually on what makes the, the app fun and unique.
18:44So that makes a lot of sense.
18:46And I love your anecdotes there of that.
18:49You don't no longer need to think about.
18:52There's this like, it's kind of like stage one, everything
18:55is crap in prototyping mode.
18:57And then at some point you need to make it real and production mode.
19:01But with react, you also don't really distinguish too much about
19:04like, Oh, am I now writing this for production or for, for prototyping?
19:09You're just doing it right.
19:10And it's easy from the get go.
19:12And if you get the same thing for data, that's super powerful.
19:16So I love that.
19:17Yeah, that's actually, that's probably a good analogy is like, How
19:20powerful, how good a quality is, is some part of your stack as if, if
19:25the prototyping is the same as like production, like there's no difference.
19:30Just another random tool I've been using lately, like SST.
19:33I don't know if you've played with that, but they have sort of their live.
19:36Development, uh, Lambda function.
19:39And it's like killer because you develop against an actual Lambda and they kind
19:43of like intercept the call to the Lambda function and send it to your like laptop.
19:48So you kind of have like debugging logs and everything, but your,
19:51everything is actually, Running in, in the cloud and it's like
19:54going to S3 buckets and everything.
19:56So it's like the development environment is the same as your
19:59production environment, um, which is like pretty hard to do.
20:01Otherwise going from dev to production with SST is like, there is no difference.
20:06Like you, it's the same stack, but the react, it's like, if your components
20:10are good for development, they're going to be efficient production.
20:12And yeah, with electric SQL, like it works in dev is going to work in product.
Schemas with ElectricSQL
20:16That's great.
20:17I would love to, to hear a bit more about the details of how you've built this app.
20:21And, um, particularly you've mentioned that you design your tables and
20:27your schemas, and you've mentioned that you're using Postgres, uh, more
20:31for like a server aspect and SQLite For, for the client state, do you
20:37design one schema and that applies to both Postgres and SQLite equally?
20:43Do you distinguish between those?
20:46Yeah.
20:46Yeah.
20:46You create a table in Postgres and then they call it electrifying it.
20:50And then there's sort of sync engine then automatically sets
20:53up that table for you in SQLite.
20:55And also like does migrations and all the other like complicated bits.
20:59You can do local only tables, but I've never needed to.
21:05There could definitely be stuff that's sort of like very specific to a client.
21:09Um, but generally speaking, like all the data, I just want synced all around.
21:13Cause you know, again, one of the cool things about all this is that, you know,
21:15your desktop, if you have a phone running the app and you have a desktop running
21:18the app, they're like always in sync.
21:20So it's, it's kind of like you pull out a phone and it's like, got the data.
Personal Apps - Pt 2
21:26So that kitchen app that you've been describing, that sounds super cool, but
21:30it sounded you've been building a few different local-first apps to really give
21:35this, give this a try and get started with, with that local-first paradigm.
21:39So which other apps have you been building?
21:41Yeah.
21:42And another app that I was pretty pleased with and I think is, uh, another
21:45good example of kind of the benefit of this like sync engine approach.
21:49It's a very simple, it's a one, one page, one view app, but
21:54basically it's like I use Garmin.
21:55I have a Garmin watch for like tracking exercise, like running, biking, et cetera.
22:00And they have a bunch of like charts and stuff that they kind of like pre designed
22:03for you, but I was like, Hey, I want to like, look at it from a different way.
22:07But basically I wanted a pure volume.
22:10Based view of my exercise.
22:11So, cause they're, they're like emphasize like intensity or mileage.
22:14And I'm just like, I just care how long I'm running or how
22:18long I'm biking, et cetera.
22:19And so I was like, Hey, they have an API.
22:22I'll just pull in the data, analyze it, create a chart off, you know, off we go.
22:26So the cool thing about it is that, so basically how the app works is like.
22:30You know, it's using electric SQL again.
22:31So it has all of your activities cached locally.
22:34So when you load it up, like it's instantaneous, it doesn't have to load
22:37a whole bunch of data off the server.
22:38It's just like runs, click SQL query presents the charts and stuff.
22:41But then also when you load it though, it like it hits a Lambda function, which then
22:46syncs in any new activities from Garmin.
22:49Into the system.
22:50And what's fun about that is like, just again, just how simple it was.
22:54Cause all it does is it like hits the API and then writes out
22:59new data into Postgres, which is like, okay, that's fine.
23:03But then the next step, you're like, well, how do you like tell the front
23:05end that like the new data is done or whatever, there's like a bunch of other
23:07stuff, but what's, what's great about it.
23:09It's like, there is no other step because the sync engines just was
23:12like, Oh, there is like a new Garmin activity pushed into the database.
23:16I'm going to push it to the client because it's subscribed to it and it's active.
23:20So it's just kind of like, it just shows up.
23:21That's again, it's kind of an illustration of like, you have this sync engine.
23:24It's not just like reactive in the client.
23:27Like there's a lot of like, we talk a lot about like client reactivity and like,
23:30there's a lot of libraries that do that, but this is more like a global reactivity.
23:33Like the whole system is reactive.
23:35Like any change anywhere, it gets pushed to every other node that cares about that.
23:40And that's just like a really powerful leap forward.
23:43Not just for the, like, Client server, but like, even like server server, like
23:47I thought a lot about like microservices, everyone's like, you know, like one
23:52big problem, microservices, they get very chatty cause they're always
23:54like, cause it's the same problem.
23:56They're always like pulling like, Hey, any new updates, any new
23:58updates, new updates, you know?
24:00And then they're like, anytime they want to do something, they're like, Oh
24:02wait, I got to go get some Jeff there.
24:03I get some Jeff there.
24:04But with the sync engine, it's like, well, no, like each microservice can just
24:07say like, this is the data I care about.
24:10And it has like a local SQLite database running that it can query against
24:14whatever it wants and like every bit of data is like up to date within like
24:18a, you know, fraction of a second that just like would dramatically speed up
24:23a lot of sort of microservices set up and dramatically reduce the chattiness
24:26going on in between different services.
Global Reactive Data
24:32Yeah, I love that analogy.
24:34I think this was really like what blew my mind about React in the, like, when I saw
24:39it the first, the first time, not just that it made my life simpler by not having
24:44to take care of like all of the dumb children, like appending, creating, etc.
24:50So it made everything simpler, but also having all of that just work
24:55automatically without me having to even like tell it like, Hey, now update.
24:59That reactivity for the views, but also for like the local state, but applying
25:04that now for global data, uh, I think that is really a killer feature of
25:09local-first that I think is not well understood by, by a lot of app developers.
25:14It's just so funny because like it's the water we live in, you know, it's like, Oh
25:17yeah, when you're building stuff, you're just, you spend a lot of time moving data
25:20around and checking it and updating it.
25:23And it's just like endless little problems.
25:25Uh, I remember like Gatsby cloud, we would like, it was such a common problem
25:30where it would like ship an update, which like, you know, use a slightly
25:34less efficient query or up the number of calls Internal API calls, and then all
25:41of a sudden, like the post, our Postgres database with like red line, like, and
25:44we like, ah, no, like, or whatever.
25:47And like, it was just like a constant headache because like, when everyone's
25:51constantly like asking for stuff, there's just enormous inefficiencies.
25:55Cause like, I remember that was a big thing where people are like early on
25:58with React, people are like, oh, like.
26:00My jQuery calls are so efficient, you know, and like react is doing
26:03all this extra work and whatever.
26:04But what pretty soon everyone kind of realized it's like, well, yes, like react
26:09does do extra work to accomplish the same goal, but it's only actually more
26:14work if every single jQuery operation was optimal, like you'd actually thought
26:21through the algorithm, you know, in detail for every single one, because in practice.
26:27Everyone was just doing lazy, weird shortcuts.
26:30You're like, you're like, I just got to ship this feature.
26:32So I'm just going to blow it away and rewrite the whole thing or whatever.
26:34He said, we were doing all sorts of like weird, crappy things to, to, to
26:38kind of like update the Dom, which, yeah, it was just like very janky,
26:43very slow all the time and whatever.
26:46So, so, so react was like, yeah, it's like, it was an
26:48overhead to doing automatically.
26:50But it was sort of like a consistent overhead and everything was consistently
26:56a little bit slower than optimal instead of like sometimes optimal,
27:00sometimes widely not optimal.
27:03And so the same thing with like moving data around.
27:05It's like, you know, people are like, Oh, yeah, like, My data fetching is so
27:10clean and so good But you know, you have one engineer who just accidentally grabs
27:16five megabytes of data from their API and like, whoops, and then all of a sudden,
27:20like the whole thing falls to pieces where, again, with like sort of more of
27:24an higher level of abstraction, it's like, you just, it just can't happen because
27:29you can even put in checks, you're like, You can only ask for max of 300 kilobytes
27:34in a batch or something like that.
27:35Whatever.
27:36I mean, there's all sorts of like.
27:38Smarter things that the system can do for you.
27:41All kinds of new abstractions of kind of like both face these same
27:45challenges of, of, or complaints.
27:47It's like, Hey, like I do it really well and you're doing it poorly
27:50in all these cases and whatever.
27:52But if the abstraction is like good enough, it eventually
27:55pretty quickly becomes.
27:56Better in most cases and often better in all cases.
28:00And even if it's not better in all cases, like taken as a whole, it just eliminates
28:04a lot of really suboptimal problems.
28:07It's like memory management, you know, that's sort of another great example.
28:10Like people are like, I manage memory so well, like these GCs, there's
28:13lots of overhead, blah, blah, blah.
28:15And then like.
28:1630 years of, uh, CVEs of, you know, security issues.
28:20It is very, very hard to build fast, efficient data loading and
28:24keeping it up to date and whatever.
28:26After like, you know, spending all this time building local-first stuff,
28:29even like very good apps built with like great teams, like, you know,
28:33name a name, like Vercel's dashboard.
28:34I mean, they're pushing like react server components and all that stuff,
28:36but like you click around their dashboard and there's like endless little, like.
28:41Loading, you know, like if they had a sync engine, they would know that the next few
28:48has all the data and it's up to date and they wouldn't have to like go back to the
28:51server, like, Hey, did anything change?
28:52Did anything change?
28:53Cause I'm sure they have a local cache, but like what
28:55they're doing is always like.
28:56Asking again on every click, they're like, Hey, has anything changed since
29:00the last time this person was here?
29:02And that, like that request, like, or sort of that, that fundamental uncertainty with
29:07an imperative approach, like, just because you just can't, you can't, you can't
29:12know if, if, if the data is up to date, um, there's no way of knowing for sure.
29:16Cause there's no system that's like reliably telling you that like, yes, the
29:20data is up to date for this next view.
29:22You have to check.
29:25And that check.
29:26Take some amount of time, which adds glitchiness into your UI.
29:29And there's just no way of, no way of getting away for that
29:31without, without a sync engine that can like provide the guarantee.
29:34I think it's really interesting.
29:36You've used the term, uh, thinking about your app data first, maybe as opposed
29:41to view first or, or react first.
29:45Yeah.
29:46I think that's kind of a big mental shift that I've also observed in my own app
29:52development evolution of my perspectives.
29:56And I think this has been a really powerful step because it intuitively
30:01makes sense that if you don't have the data, then if you want to render it.
30:06And you don't have it, then you also can't render it.
30:09And so thus you render a loader or a skeleton, et cetera.
30:14Whereas if you think about your app.
30:17Navigation routing user experience overall more in a data first way to
30:22think about, okay, I'm currently here.
30:24This is all the data that I need, but also think a step ahead of like, Oh, what if
30:29I would go to, to this other route here?
30:32Maybe I should already have the bare essentials in terms of the
30:36data for that, that if I'm going there, it's immediately available.
30:40And then you think about your data more, maybe you think about it as a
30:44graph or like a collection of documents, however, you want to think about the
30:47data, how it fits your app's use case.
30:49But once you start thinking about your app data first, I think it's
30:54so much easier to build a really.
30:56high quality, fast user experience then, um, and it's also so much
31:01easier to make that available offline.
31:03I'm not sure whether you need Vercel's dashboard, which is gorgeous, but has
31:08a bunch of loaders, loading spinners.
31:09I don't think you need that necessarily to work offline.
31:12But another way to, if it works on offline, then it's crazy fast.
31:17And I think it would be great if it was a lot faster.
31:21So I would, I would completely agree with kind of thinking about your, about
31:25your app development workflows as data first, as opposed to React or Vue first.
31:32So we've been talking a lot about the, all the good things now that you've
31:36experienced with local-first, but I'm sure you've also experienced like
31:40some pain points, some challenges, um, in regards to that new approach
31:45of building software or in regards to the technologies you've been using.
31:49Tell me a bit about that, which sort of challenges have
Learning curve & other challenges
31:53Yeah, there's definitely a few.
31:54I mean, there is some learning curve.
31:56It's not the same as what was before there.
31:58There's a lot of things that cross over.
32:00So it's not like a huge step.
32:02You're going to have to like spend some time learning stuff regardless
32:06of whatever tool you choose.
32:07You're gonna have to like learn stuff for that that you don't know now.
32:09But I think there's also at this current point of immaturity in the market,
32:13there's not like a clear market leader.
32:15Yeah.
32:15Yeah.
32:16Who's very good and mature that lots of people use.
32:18So, you know, like in in, in sort of we're like, Hey, next Js.
32:22Like that's sort of the obvious choice.
32:23'cause the billion people use it and it, it's kind of
32:25covers everything and whatever.
32:26So you both have the, I have to evaluate a bunch of tools, a
32:29step to get, get into this world.
32:31And then you're like, and then I have to learn a specific tool in depth.
32:35'cause there's, there's, it's just different than, you know,
32:37like no one's done it before.
32:39Almost nobody's done it before.
32:40So you're going to have to like learn stuff to do that.
32:41So that's definitely a bit of an overhead, uh, in this approach.
32:45And another result of the immaturity is like, there's just things that don't work.
32:49So there's sort of the magical, like I'd be kind of describing like sinking,
32:53just like, Oh yeah, they're just sort of magic, but they're not actually magic.
32:56They actually.
32:57are an engineered product and there's stuff that they just don't do yet,
33:02um, that you'd rather like them to do.
33:04And so it's, uh, you know, learning those limitations and like working within
33:08them or even, you know, even decide if you can or not is part of sort of the
33:12evaluation process and building process.
33:14So far for my toy apps, it hasn't been an issue, but like, there's definitely
33:17things that I'm like, I can't build those things with this stack right now.
33:21And like, yeah, the engineering, you know, the teams behind
33:22these, like they know about it.
33:23And like, they're working hard to, to, to fix them.
33:26A lot of stuff is kind of like just the same.
33:28It's just building a map.
33:29You're just like, I need data and data's there.
33:31Before I'd write a REST API call or GraphQL call or whatever.
33:35And on the backend, I'd write a query.
33:37And now you just write the same query, but it runs locally.
33:41And the day to day, just like building views and stuff, it's extremely similar.
33:46You know, it's like before you'd probably like to sort of hash out like
33:50the, the data locally and the client.
33:52But now I just started like, Oh, I need to like change my data model.
33:54So I'll just tweak the, you know, the database structure a
33:57bit and then, you know, change my query a bit in the front end.
34:00So yeah, I mean, day to day, it's like, it's very similar,
34:03but just like less steps.
34:05Cause you don't have to like change your GraphQL or you don't have
34:08to change like the client call.
34:10It's only like two steps.
34:11It's like you change the database table.
34:13And then you change your query in, in, in your, in your component.
34:17And then that's about it.
34:18So it's pretty, it's pretty simple.
34:21That makes a lot of sense.
34:22And I mean, just overall trade as for, for most early technologies, if you're
34:27early to those technologies, you bet on some superpowers and you probably
34:32acquire those new superpowers earlier than most of us, but you also got to
Technical pioneering
34:40Yeah.
34:40It's just like, yeah.
34:41I think the biggest thing is like.
34:44I just randomly have gone into like, Oh, do, do, do, do, do, I'm doing something.
34:48And then like, bam, I hit something and then I'm like stalled for like
34:51a day while I either figure it out or, or figure out a workaround.
34:56So I think it's, it's, it's kind of like if you're using boring technology.
35:00It's like, it may be tedious and hard or whatever, but it's
35:04sort of like a known tedium.
35:06You're like, this is going to take me like one week, where with a new
35:10technology, it's either going to take you an hour or two weeks, you know?
35:14So, uh, it was a lot more uncertainty on what's going to happen.
35:18And you might run into stuff that just like absolute blockers where again,
35:21with the old stuff, it's kind of like every single possible work around.
35:25Has been figured out for every possible thing.
35:27And so it's just a matter of finding that obscure forum post from like
35:31five years ago, where they're like some weirdos, like, Oh, I spent five
35:34days on this, but here's the fix.
35:35And you're like, freak.
35:36Like we are, we have a kinship.
35:38You and I know we two people are the only ones that have run into
35:41this problem before, but thank you.
35:43You, you wrote it down for me, so I don't have to figure it out myself.
35:45So, so yeah, the new technology, it's like most of the problems you run into, you
35:49are the very first person to run into it.
35:51Which is fun.
35:51It's sort of like, yay, you're like, I don't know.
35:53But yeah, it's, it's, you're gonna have to be like a pioneer, sort of a
35:56more robust, I don't know if Europeans talk about pioneers, but anyways,
36:00like American pioneer, you're kind of like, ah, we're going off into
36:02the wilderness and got an axe on our shoulder and, you know, whatever.
36:06We're just gonna like make it happen.
36:07You kind of have to adopt that, that mindset, um, when you're
36:11kind of into new technology.
36:13Yeah, that's very relatable for me.
36:15I've always considered myself to be kind of like a technical pioneer.
36:19So what you've just mentioned in terms of, yeah, sometimes like it's a gamble
36:23and sometimes like it's the, the sort of promise of everything will be simpler
36:28and easier in the future that plays out.
36:30And you can't believe like how the thing that just took so much effort
36:34and so much work in the past is now so easy, or you run into a wall Or
36:40in theory, it would have worked, but you run into some edge case that
36:44you're the first one running into.
36:46And it might be fixed with the next release of whatever you're using,
36:50but you still have run into it.
36:52But luckily you've run into it.
36:54And then the next person is no longer running into it, depending
36:58on how quickly someone releases.
37:00Yeah, there's there's sort of a sense of community service you, you, you.
37:04Get, or at least it's helpful to get with new technologies.
37:06You're like, yeah, I'm going to pay down the cost for everyone else a bit.
37:10So another observation here, I think it's really interesting that
37:14the use cases you've mentioned, your ambitions for those apps.
37:20We're not to like take it to market and like roll it out to millions of users.
37:24It was, I think like part of the fun that you built a software
37:28that's just for you and you can make it as fancy or ugly as you want,
37:34like no, one's going to complain.
37:35and you know like the little quirks, uh, of it and that it took you less effort to
37:41make that happen, to build the thing you wanted to build and like spend less time
37:46with like all of the boilerplatey things.
37:48That's great.
37:49And I think that, that is totally fine.
37:51Like no one's going to take you, take that away from you.
37:55Uh, I think what's still a little bit less proven is like, how do you
37:58really If you were to take that app now and say, I do want to scale it
38:03to millions of users, et cetera.
38:04I think that's another aspect of where we are still pretty early.
38:09And on our pioneering journey, we have not yet figured out like hit all the traps.
38:15But, uh, I think from first principles, I think, uh, this
Developing, deploying, scaling
38:22I don't, I don't see.
38:24Any like showstoppers for sort of scale to millions and a lot of ways again too.
38:29It's like, it's actually no apps do scale to a million.
38:33What happens is people build it and it falls apart and then they like rebuild
38:37it several times and get to millions.
38:40So what actually I think is actually pretty interesting about this and like
38:42promising is like there is a story here prototype to millions with like
38:47no changes to your basic architecture.
38:51Which I don't think anything else can make that claim, but like the base system of
38:56like what everyone plans to do is like you can horizontally scale sync engines
39:00without any issues because you can just have sticky, sticky connections from
39:04clients to, to particular sync engine and it's managing the state for each user.
39:07Postgres is easy to vertically scale.
39:10What's interesting else, but like Postgres actually has like a lot
39:12less demand in this world because far fewer queries are reaching it.
39:15So it's actually like you don't have to scale Postgres nearly as fast as
39:19you kind of typically expect to, you know, with this approach, people either
39:22do sort of normal API stuff because you can still just call Lambda and
39:25do whatever and write to Postgres.
39:26Can I was talking about the Garmin example, you can write locally, like,
39:30I want to do this sort of our sort of event sourcing style where you're like,
39:33the client says, I want this to happen.
39:35And then the back end.
39:36Serverless function picks up the job and then kind of writes
39:39it back and then it gets synced back to the, the client anyway.
39:42So that, that, again, that approach is also extremely scalable because it's
39:46all serverless and stateless and you can have as many workers, you know, spinning
39:50up to, to handle stuff for people.
39:51But yeah, we're not there.
39:52This is all very, that's all very potential.
39:54But I think what's extremely promising is like, you know,
39:57what are the different stages?
39:58What are the three major stages?
40:00It's like how efficient is a tool of prototyping?
40:02How easy is to go from prototype to production and
40:05then how easy it's to scale.
40:06So those are like three distinct problems.
40:08So problem line prototyping, I think both of our experiences, it's a lot
40:13easier to prototype with a stack.
40:14Like it's data centric, you have like a full type safe client, you know, from
40:18the get go, a lot of boilerplate is gone.
40:20Like you just like, bam, you go.
40:22And then again, like prototype production, extremely easy.
40:25Like I, like I've done lots of side projects in the past that I
40:28always kind of came down to it.
40:29Now I'm going to.
40:30Deploy it.
40:31And I was like, uh, I have to rewrite so many things and I have to like, do all
40:36this crap that I didn't have to do before.
40:37Like, Oh no, where's the steak going to go?
40:39Whatever.
40:39Like I I've gotten, I've gotten my sort of go live checklist.
40:45It's like five things and like, it takes me less than an hour.
40:49So it's extremely fast to go from like prototype to production.
40:53And then, you know, production scaling.
40:55Yeah.
40:55That's, that's, that's the part that doesn't really exist, but in theory,
40:59like, I think the architecture is like.
41:01A very scale free, like, it's mainly just a sync engine that, um, needs.
41:08Proven at scale.
41:10I mean, I do think that I fully agree with the way how you delineated
41:15that into those like 3 steps.
41:17Um, but I would even say that the scaling part I think has been proven
41:22by quite a couple of companies by now, whether, uh, Whether it's like thinking
41:26about Replicash, uh, Rosicorp has built with Replicash and Reflect, et cetera.
41:31I think they have a couple of customers that are really going at scale and
41:35the entire system I think is quite inherently scalable and, uh, the, it
41:41doesn't change the, the simplicity of the system, which is, which is the beauty.
41:46Um, and a lot of, a lot of the, a lot of the parts of the system can
41:50still be considered stateless, which makes it a lot easier to reason about.
41:54linear, linear had a really great talk about, uh, linear
41:58CTO, uh, had a really great talk about, uh, how, how they scaled.
42:04And it's, it's pretty fascinating, like all the things that they
42:06did, but yeah, to your point, people have gone down the path.
42:10So, you know, innovation kind of goes through, like,
42:13is this thing even possible?
42:15Or somebody has to, like, discover something.
42:16Like, there's this like, oh, there's this new thing that we can do.
42:19Wow.
42:19Who knew?
42:20And then it's sort of like.
42:22It goes from that to then sort of eventually to sort of a product
42:26where it's kind of like that technology is then encapsulated
42:29to something that anybody can use.
42:30So the fact that multiple independent teams have successfully gone down
42:35this path shows that then sort of now.
42:38What we're seeing is sort of a productization of sync engines.
42:41If you want to do a sync engine, you have to like buy it.
42:42You don't have to build it.
42:43You can just buy it off the shelf.
42:44Um, I think we're pretty close to that being a reality.
Approaching local-first
42:48So you've been going through this journey, I guess, over the last
42:52year or so of like learning about local-first and step by step building
42:56your own apps with local-first.
42:58What would be your advice to an app developer who's curious
43:02about local-first, but has not yet taken their own first steps?
43:06What would be your advice to them?
43:08I don't know.
43:08My, my, my general advice on any technology is just to dive right in.
43:12I mean, there's lots of stuff you can read about it.
43:14So, so by all means, like, you know, listen to cool, cool podcast.
43:18There's lots of blog posts.
43:20Yeah.
43:20So it was definitely lots you can read and listen to and watch and stuff.
43:23But, um, there's no better way to feel it, to like, to understand
43:26it and to really feel it than just to like start building some stuff.
43:29All the different tools have starters and they're very approachable.
43:33They're not like particularly hard to use.
43:35And, you know, Take a day or two and build something.
43:39Cause yeah, it's really just like, when you just see it sinking between
43:43tabs and you're just like, you're just kind of like, you, you really
43:47kind of have to feel whenever you're just like used to doing something.
43:50You just, you just forget about it.
43:52We get numb to all the stuff we do.
43:54So you really just have to like.
43:55To really feel the difference of local-first you just have to build
43:58something and then you're just like where's all this stuff that you just sort
44:02of feel voids in your in your soul you're like you're like where's all the missing
44:06stuff you know like where's all those things I normally do and you're just kind
44:10of like oh they're just gone and then like after you've done a few you're kind of
44:13like You just start to forget about it.
44:15It's like this bad, weird dream of like spending all that time on all
44:18those things that are now gone.
44:20But yeah, you just, you can't really appreciate that until
44:22you've actually built something.
44:24Uh, let's see whether in five years from now, we look back into the days where
44:28we haven't built, been building with local-first and we see it just crystal
44:32clear as that analogy you've given before.
44:35With going from jQuery to React and React freed up, uh, freed us up
44:40from having to think about how to render the views in the correct way.
44:45And now we get the same thing for data.
44:48So this is beautiful.
44:50Now, yeah, talking, talking to younger developers, like sometimes
44:53I mentioned jQuery and I'm like, Oh wait, they've probably don't jQuery is.
44:59And it's just a very weird feeling to be like, Oh, yeah, just jQuery
45:02thing it was like this and they're just like, yeah, okay, whatever like
45:05yeah It'd be it'd be really amazing if like apis was the same thing.
45:08Like yeah, we used to like You'd write like an HTTP method and then you sort
45:15of like make up a name that sort of meant something and then you like how
45:20to like remember to like write it all the same thing and your client you know
45:24you're like it's not a post it's it's a put and you have to like no typos and
45:29they're like that's it's like a lot of tedious, fragile, annoying code to write.
45:35And you're like, yeah, it was, it sucked.
45:37And they're like, I just used like the library and I just sort of typed stuff
45:40and like, you know, Copilot v10 just sort of like spits out whole, whole, whole
45:45stuff, you know, cause it just introspects on my whole data system and like writes
45:48out super efficient stuff for me.
45:49And yeah, I, I, I'm very big on like, you know, getting rid
45:53of work we don't have to do.
45:55Um, and I think there's a lot left in programming that you
45:58really shouldn't have to do.
Outro
46:00Let's leave it here.
46:01Anything you want to share with the audience?
46:04Have fun, build cool stuff.
46:06That sounds great.
46:06Kyle, thank you so much for coming on the podcast.
46:09This was a lot of fun.
46:10Yeah, it was a lot of fun.
46:12Thank you for listening to the localfirst.fm podcast.
46:15If you've enjoyed this episode and haven't done so already, please subscribe and
46:18leave a review wherever you're listening.
46:20Please also tell your friends about it.
46:22If you think they could be interested in local-first, if you have feedback,
46:25questions or ideas for the podcast, please get in touch via hello at
46:29localfirst.fm or use the feedback form on our website, special thanks to Expo and
46:35Crab Nebula for supporting this podcast.