Sponsored by
Show notes
Transcript
0:00:00 Intro
0:00:00 Well, can I just say that, you know,
from the point of view of exhaustiveness,
0:00:04 there are many, many unsolved
problems in the local-first space.
0:00:09 And I have a horrible feeling
that it's easy to think that
0:00:13 synchronization is the problem.
0:00:14 And then once you've solved
synchronization, you're good.
0:00:17 Well, I don't think that's true.
0:00:20 I think we all got hung up on, oh,
once we crack CRDTs, we'll be good.
0:00:23 no, no, no, no, that's the easy part.
0:00:25 It's not state management, it's the
state of the state management in a way.
0:00:28 It's like, should I be
synchronizing this data?
0:00:31 Because this person isn't logged in,
or is logged in, or is online, or is
0:00:35 offline, or has shared this document,
or hasn't shared this document, and
0:00:39 that is actually unsolved, I think.
0:00:42 Welcome to the localfirst.fm podcast.
0:00:44 I'm your host, Johannes Schickling,
and I'm a web developer, a
0:00:47 startup founder, and love the
craft of software engineering.
0:00:51 For the past few years, I've been on a
journey to build a modern, high quality
0:00:55 music app using web technologies.
0:00:57 And in doing so, I've been falling down
the rabbit hole of local-first software.
0:01:01 This podcast is your invitation
to join me on that journey.
0:01:05 In this episode, I'm speaking to James
Pearce, the author of TinyBase, a reactive
0:01:09 datastore library for local-first apps.
0:01:12 In this conversation, we explore
how TinyBase works, including its
0:01:16 custom query system, the various
persistence and syncing integrations,
0:01:20 as well as James plans for the future.
0:01:22 Before getting started, also a
big thank you to Rosicorp and
0:01:25 Expo for supporting this podcast.
0:01:28 And now my interview with James.
0:01:31 Hey, welcome James.
0:01:32 Nice to meet you and nice
to have you on the show.
0:01:34 How are you doing?
0:01:35 I'm doing great.
0:01:36 Thank you so much for having me.
0:01:37 Huge privilege to have a
chance to come along and chat
0:01:40 about all things local-first.
0:01:43 Facebook/Meta
0:01:43 It's safe to say that you're the
first guest so far in the podcast,
0:01:47 who's calling in from a boat, which
I think is very on brand for what
0:01:52 local-first software should enable.
0:01:55 and I think Maybe there's a sort of
romantic pitch for why you're doing
0:01:59 local-first, or, or maybe not so much.
0:02:02 I think we'll get to that in a moment,
but the first time, I've took notice of
0:02:07 you is like when you've been at Facebook
back then taking care of overseeing a
0:02:12 whole bunch of open source projects.
0:02:13 So would you mind taking a step back,
briefly introducing yourself and sharing
0:02:18 more about, your time at Facebook?
0:02:20 Sure.
0:02:21 Certainly.
0:02:21 So, I'm James.
0:02:22 I did have a career before Facebook
slash Meta, fairly long time ago.
0:02:27 I'm from the UK originally.
0:02:28 I've been in tech nearly
all of my professional life.
0:02:31 Moved to California in about, 2010, and
started work at Facebook and worked on a
0:02:38 variety of things during my time there.
0:02:40 I was at the company for just over
10 years, but, you know, one of the
0:02:44 highlights for sure was working on.
0:02:46 The open source program, as I'm sure
most of the listeners are aware Meta
0:02:51 does have quite a large investment in
open source whether it's on the client
0:02:55 JavaScript technology side or on DevTools
or on data infrastructure and of course
0:03:02 now on AI, ML and LLM technologies.
0:03:07 And so yeah, it was a, an
awesome period of time for me.
0:03:10 It was between about 2014 and 2017.
0:03:14 So I helped out with a bunch of
the launches of things like React
0:03:18 and PyTorch and HHVM and various
build tools and a whole bunch of things
0:03:24 that people are probably aware of.
0:03:26 And yeah, it was a super exciting time,
just in terms of helping to develop how
0:03:31 the company thought about open source and
what open source could do for the company.
0:03:36 Wasn't the only thing I did
whilst I was at Facebook.
0:03:38 Also worked on Portal, which was the
smart speaker device that the company had.
0:03:43 , sadly now discontinued, but, it was,
that was pretty awesome working on,
0:03:46 on a hardware project for a bit.
0:03:48 And also spent some time in the data
infrastructure team, working on some of
0:03:52 the large data warehouse technologies,
that also Meta is fairly well known for.
0:03:57 So, yeah, one of the great things,
maybe underappreciated, features of
0:04:01 being an employee at Facebook is you,
you do get to, move around a lot.
0:04:04 There are lots of mobility opportunities,
and I took the greatest advantage of
0:04:08 that, that I could, and worked on a
wide, wide range of different things.
0:04:11 but yeah, enjoyed, every minute
of it and then in about 2022.
0:04:16 So yeah, just coming up on two years
ago, figured it was a, a good opportunity
0:04:20 to try something completely different.
0:04:21 And so, Moved on, moved on board Scout,
which is the boat that you see me on now.
0:04:27 And, sold up, my house
and life in Silicon Valley.
0:04:31 And I've been sailing on the ocean
waves ever since writing open source
0:04:35 software, so that's where we are.
0:04:36 That sounds quite incredible.
0:04:38 And I mean, Facebook is certainly.
0:04:41 One of the companies that comes to
mind is like how to have a positive sum
0:04:46 vision on how to do open source at scale.
0:04:49 , I can't really think of another
company that has had such an, such a
0:04:53 positive impact on our web ecosystem.
0:04:56 and in particular With
React, with GraphQL.
0:05:00 I'm not as deep on the AI side,
but I mean, the recent releases,
0:05:05 et cetera, around Llama, et cetera,
this is all, I think a great example
0:05:09 of how a large company can have a
very positive impact on the, the
0:05:15 ecosystem beyond its own benefits.
0:05:17 So it's great to hear that you must've
had like a very positive impact on that.
0:05:22 The open source culture at Facebook
0:05:22 Yeah.
0:05:22 It's a lot of this.
0:05:23 took time to develop.
0:05:24 I mean you could argue we stumbled into
some of these benefits inadvertently.
0:05:29 I think early on it just seemed
like the right thing to do.
0:05:32 The company had been built on other
people's open source when, you know,
0:05:36 Mark Zuckerberg started the site.
0:05:38 PHP, running on Apache with MySQL.
0:05:41 And so there was always this sense of
standing on the shoulders of other giants.
0:05:44 And so there was a, an almost cultural
obligation to share back things like
0:05:49 Memcached and, you know, some of the early
things that the company was working on.
0:05:53 but then it began to become apparent that
it was more than just that obligation.
0:05:57 It was also, starting to help lead
the industry because as Facebook
0:06:01 grew, it was experiencing problems
that other apps or other sites
0:06:06 weren't, just because of its scale.
0:06:07 And so it was like the advanced guard
of finding out what happens when you
0:06:11 build really complicated web apps.
0:06:13 and so that was what gave birth
to React, you know, at the
0:06:16 time that React was launched.
0:06:18 You know, most web apps were still
MVC that was kind of the prevailing
0:06:22 pattern and it was still, still super
easy to have inconsistencies between
0:06:27 state across a big complicated app.
0:06:29 And so, you know, React's innovation
was, you know, how do we make sure that
0:06:32 the, the user interface is always a pure
function of data that's gone into it.
0:06:37 And, you know, at the time it seemed
heretical, but pretty quickly other
0:06:40 people started to hit the same.
0:06:42 scaling issues with, with MVC
apps and, and, and realized
0:06:45 that React was maybe a way out.
0:06:46 And that's the pattern I think at Facebook
over and over again, whether it's data
0:06:50 infrastructure or or even ML, you know,
the, the company is experiencing problems
0:06:55 that others will face, but haven't yet.
0:06:57 And so it has a chance to maybe share back
some of its solutions from the future, so
0:07:02 to speak, so that others then benefit from
those as they experience those challenges.
0:07:06 Um, so that was, that was one
of the big drivers, I think.
0:07:09 Obviously, it helps
with recruiting, right?
0:07:10 Because you're demonstrating the
kinds of problems that you're solving
0:07:13 and that gets people excited to
come and help solve those problems.
0:07:16 and it also kind of opens the windows
onto the kind of code that people are
0:07:20 building at a company like Facebook.
0:07:22 And obviously, there are lots of world
class engineers there, and there's no
0:07:26 better way of learning that than seeing
the kind of code that they're writing.
0:07:29 , and so it kind of gives you that little
bit of visibility into the, the caliber of
0:07:34 the, the engineers that are working there.
0:07:35 And yeah, so there, there were
benefits coming back to the company
0:07:39 and just in terms of engineering brand
and recruiting, that over time became,
0:07:43 that became so obvious that it was
like, okay, now we need to just keep
0:07:45 investing in all of these things.
0:07:47 Um, and it's now more than just
a thing that we're doing for fun.
0:07:50 It's now something that's actually
bringing real value to the business.
0:07:54 Exactly.
0:07:54 And I think there's no coincidence that
now some of the most prolific, open
0:08:01 source and tool builders, actually used
to work at Facebook, like thinking of
0:08:05 the people who invented GraphQL, now
building other technologies or Matt
0:08:10 Wonlaw, who was just on the show, was
also at Facebook for, for a while.
0:08:15 So I think really like this deep culture.
0:08:17 around great developer experience
and very principled thinking,
0:08:21 like marrying those two together.
0:08:22 I think this is what, what makes
tools really successful and sticky.
0:08:27 And I think this is what the internal,
Facebook engineering culture has
0:08:32 sort of like brought to light.
0:08:34 I will say it's not as easy as it sounds.
0:08:36 there are lots of things you have
to do behind the scenes and as
0:08:38 someone that was responsible.
0:08:40 for the the overall program rather
than individual projects, you know, I
0:08:44 needed to think about how we I guess
sell this concept internally, right?
0:08:49 Because if you're an engineering manager
and you're running a team and they've
0:08:53 decided they want to spend 20 percent of
their time supporting their project on
0:08:57 github You're like wait, but now i'm down
20 percent in terms of like my internal
0:09:00 goals So a lot of it was a lot of the
challenges were internally Making sure
0:09:05 that everybody was on board with the,
the macro global benefits, even if at a
0:09:09 local level, it seemed like it was a cost.
0:09:12 but we got around that in a
variety of different ways.
0:09:14 , one of the ways that, you know,
perhaps isn't obvious, but I will
0:09:16 state it because I think it was, it
was perhaps even innovative at the
0:09:20 time, is make sure you're using exactly
the same code from GitHub internally
0:09:25 and I see lots of companies that,
you know, are doing open source, but
0:09:28 it's a fork that they've thrown over
the wall from a couple of years ago.
0:09:31 And now they've diverged and they're doing
something completely different internally.
0:09:34 and that never works because of course
that always gets forgotten about relative
0:09:38 to the thing that's being used internally.
0:09:39 But I think it's pretty awesome that
you go to GitHub and you look at slash
0:09:44 react, and then you go to facebook.
0:09:46 com and you view source.
0:09:48 And you will see exactly the same code,
because the, the internal version of
0:09:52 React and the external version are
being synchronized commit by commit.
0:09:56 and so that means that
you never get out of sync.
0:09:58 It means that the engineers who
are working internally get to
0:10:01 see their value immediately going
to the community and vice versa.
0:10:04 The community gets to see their value,
appearing on the world's largest website.
0:10:07 So, those things are hard, like,
infrastructurally to get working, but
0:10:11 those are the kinds of things that, that
then help the whole open source machine
0:10:15 work inside a large company like that.
0:10:18 How James got into local-first: living on a boat
0:10:18 So in 2022, you mentioned you sold your
house, you, like, made a drastic change to
0:10:24 your life, and you moved onto your boat.
0:10:27 how Did that lead you to local-first?
0:10:29 It seems to coincide a bit timing
wise to, to also, start like working
0:10:35 on TinyBase as your primary project.
0:10:37 Can you share more about that?
0:10:39 Yeah, I guess with the benefit
of hindsight, you can rework the
0:10:42 origin story a little bit and say,
Oh yeah, it was all intentional.
0:10:45 it wasn't, of course.
0:10:46 I had.
0:10:47 Some good milestones in my life you
know, my, my kids were leaving home and
0:10:52 had reached my 10 year anniversary at
Facebook and the boat that we had been
0:10:57 dreaming about, you know, was, was ready.
0:10:59 and so it seemed like it was a good idea
to do all these, these things at the same
0:11:02 time in terms of my, my life choices.
0:11:05 but as regards local-first and TinyBase,
which is the project that I've been
0:11:10 working on that I'm sure we'll, we'll talk
about, that was somewhat Coincidental,
0:11:15 that was something that I had been, you
know, thinking about for a long time.
0:11:18 but I think, yeah, then once you
move on a boat, the, a number of the
0:11:23 benefits of, of building local-first
apps suddenly become acutely clear.
0:11:27 It's like, wait, Oh, sometimes I
really just don't have a cloud.
0:11:30 And sometimes even if I do,
my network is pretty bad.
0:11:34 And it really throws into perspective
the difference between the apps that
0:11:38 I native apps, right, that do have
much better local support and the web
0:11:44 apps, which just now fail to load.
0:11:46 And it really threw into stark relief
the difference between them at the
0:11:50 same time, as I said, I was working on
TinyBase and maybe I can reverse the local
0:11:55 local-first principle into into the origin
story of TinyBase because it does seem
0:11:59 to help support those kinds of things.
0:12:01 and then like almost within a month,
I then read the Ink and Switch article
0:12:05 and I saw the essay about Rffle.
0:12:07 I'm like, okay, all my, all these
things are converging and , and
0:12:12 it all suddenly made sense.
0:12:13 So there was a lot of serendipity
involved in kind of weaving
0:12:17 these narratives together.
0:12:18 But yeah, certainly life on a
boat does, does make you think about
0:12:23 local-first a lot more than living
in some, you know, high density,
0:12:26 high network connectivity lifestyle.
0:12:28 So, yeah, no, it definitely helped inform
my thinking about this and has encouraged
0:12:36 The journey to TinyBase
0:12:36 So you mentioned keep working on TinyBase.
0:12:38 So something must've led you to
start working on TinyBase to begin
0:12:43 with, and later you oriented more
around local-first, what was the
0:12:48 impetus to start working on TinyBase?
0:12:50 So, well, maybe I should just
like rewind a little bit in case
0:12:54 it's not blindingly obvious.
0:12:56 I love building stuff and all of
my life, I guess, I've been, you
0:13:01 know, surrounded by computers.
0:13:02 I was programming when I was a kid
and, you know, grew up always trying to
0:13:06 create things with code and I guess the
lifestyle of a manager or a director at
0:13:12 Meta is that you tend to get a little
further and further away from the code
0:13:17 base, which I found sad because even
though I love managing teams and people
0:13:22 and helping grow teams, Organizations
and so forth, like some part of me,
0:13:27 like a primeval part of me still needs
to be building something of my own.
0:13:31 And so, that, that was really the origin
of just doing something, anything.
0:13:36 I just needed to be able to
keep building even if it was
0:13:39 just a small project for myself.
0:13:41 And then obviously once I left the
company and I started to find I had
0:13:44 more time, I was able to start investing
a little bit more effort into it.
0:13:49 And yeah, the, the original kind of story
for, for building TinyBase was that I
0:13:55 had an idea in my mind of an app that
would be really cool, but I knew that
0:13:59 in order to build that app, I was going
to need to have something like TinyBase.
0:14:03 So, you know, without going into too
much detail, you know, when we were
0:14:06 at when I was at Facebook, I, I, I
needed a way to manage the state of
0:14:11 hundreds of open source projects, right?
0:14:12 And so we built some internal tools
that allowed us to, Pull from GitHub, you
0:14:18 know, the state of all of the repos, how
many pull requests, issues, and so forth.
0:14:22 And um, That tooling really made
it easy for us to look at this
0:14:26 huge portfolio and figure out what
was going well and what wasn't.
0:14:29 But I felt like the world itself
could benefit from something like
0:14:33 that So I thought maybe maybe there's
a there's an app out there somewhere
0:14:36 which would be , a GitHub at scale
kind of interface that would let you
0:14:40 manage your open source portfolio.
0:14:42 And so anyway, I thought, well, if I
needed an app like that, and I'd need
0:14:45 to put it in a browser, and I'd need
to have some kind of way of storing
0:14:49 the tables of repos and issues and pull
requests, you know, how would I do that?
0:14:53 Like, what is a good tabular data
store that runs in the browser?
0:14:57 And there isn't, well, back then
anyway, 2021 2022, there wasn't
0:15:02 really an obvious way to do that.
0:15:04 And so I thought, well, rather than
building the app, I'll start off by
0:15:07 building a state management library that
allows me to store tabular data that in
0:15:13 the future will store this data that I
need in this app that I haven't built yet.
0:15:16 And yeah, that, that, that's
where TinyBase originally came from.
0:15:20 I knew it needed sort of database
semantics, but didn't actually
0:15:24 need to be a database, right?
0:15:25 It could still be considered
an in memory data store.
0:15:28 and I'm sure everyone's familiar
with the idea of a, you know,
0:15:31 kind of a store in a React app.
0:15:33 But I wanted it to kind of look and
feel a little bit like a database.
0:15:36 I wanted it to feel like it
had tables and rows and cells.
0:15:39 I wanted to do queries against it, and
I wanted to be able to do aggregates
0:15:42 of the data so I could count up
the number of stars or commits in a
0:15:45 repo and all those kinds of things.
0:15:47 so yeah, that was the original, idea
of TinyBase, you know, something
0:15:51 that wasn't a database, but sort of
had database conceptual semantics.
0:15:56 and that's where it popped up.
0:15:57 And then of course I stumbled
into the Rffle And you
0:16:01 know, Ink and Switch material.
0:16:03 And I obviously spotted that you
were doing something similar with,
0:16:06 with a SQLite approach, which is
obviously literally a database.
0:16:09 So you literally get
the database semantics.
0:16:12 but yeah, by that point I'd
kind of committed myself to
0:16:14 doing an in memory approach.
0:16:17 So I thought, well, let's, let's try both.
0:16:20 , and I, and so I kept pushing,
pushing forward with TinyBase.
0:16:23 Now the story has obviously
evolved a lot since then.
0:16:25 TinyBase is now a lot better about
integrating with real databases.
0:16:29 We can come on to talk about that.
0:16:30 Um, but yeah, that was,
that was the origin story.
0:16:33 It was just something I needed for myself.
0:16:35 I thought the world needed,
and I went and built it.
0:16:37 And then once I found I had more
time, I guess I kept building.
0:16:41 that is amazing.
0:16:41 I think there's a lot of similarities
in terms of your motivations, et cetera.
0:16:46 Maybe a few different technical
decisions along the way, but for me,
0:16:51 starting to work on Overtone would
have not been feasible without having
0:16:56 a very substantial State, local state
management foundation to build on top of.
0:17:01 And this is how I then like got
to know Jeffrey, et cetera, and
0:17:05 started getting involved with Rffle.
0:17:07 So it seems like there is a clear parallel
to you wanting to build your own app
0:17:12 and realizing, okay, that's going to
be terrible working on that app without
0:17:17 those primitives that you have in mind.
0:17:19 So obviously you got to start
building your, your own state
0:17:22 management system first.
0:17:24 so, and kudos to you.
0:17:26 I think TinyBase is one of the most
polished and, sophisticated solutions
0:17:31 that is already out there where I think
a lot of technologies are, maybe have
0:17:37 really great ideas, but are still.
0:17:40 pretty early in terms of being
approachable and having the
0:17:43 supporting resources around it.
0:17:45 So, for example, LiveStore
is not yet open source.
0:17:48 It's available to GitHub sponsors,
but it doesn't have documentation yet.
0:17:52 It doesn't have guides yet.
0:17:53 It doesn't have, like, all of those,
supporting material around it.
0:17:57 Whereas TinyBase has a gorgeous website.
0:18:00 Really polished docs, lots
of examples, lots of guides.
0:18:05 you keep, releasing regularly,
like new versions, integrations
0:18:09 with different other technologies.
0:18:11 So really, really exciting
how far you've come.
0:18:14 I'm now looking forward to trying to
understand better the different details.
0:18:18 Right.
0:18:19 Well, just firstly, a bit, a
little bit of a meta point on
0:18:22 the, on the kind of the polish.
0:18:23 so a side kind of motivation for
this was to see what it was like
0:18:28 to build software where I didn't
have to make any major compromises.
0:18:33 and I'm sure you and, and, and the
people listening or watching, you
0:18:36 know, have, have written software where
you're under a deadline or there's
0:18:40 some set of constraints and you just
start cutting corners, you know, ah,
0:18:44 you know, this bit of code is really
crappy or, well, we didn't have time
0:18:47 to finish the test coverage or, no,
just, we've got to ship it, right?
0:18:50 Cause that's, and actually that's fine.
0:18:52 Right.
0:18:52 I mean that, that, that's life.
0:18:56 I always feel like some.
0:18:57 Sort of perfectionist part of my brain
is always like, ah, like next time,
0:19:02 next time, I'm going to make it perfect.
0:19:03 Next time, you know, the
architecture is going to be perfect.
0:19:06 Next time the test coverage is
going to be a hundred percent.
0:19:09 And so given that I now had this
project that only I was working on,
0:19:13 I was really only my only customer.
0:19:15 I had a bunch of spare time.
0:19:17 I figured.
0:19:18 , if I'm ever going to have a chance
to build, quote, what I thought was
0:19:21 perfect software, and of course that's
subjective, it's just my view that it's
0:19:23 perfect then this was going to be it.
0:19:25 So, can I build a piece of software that
has, you know, 100 percent test coverage,
0:19:30 is refactored, like, so iteratively that
eventually it can't be refactored anymore.
0:19:36 , it's got all the documentation
you could ever need.
0:19:39 It's got the best website that me
and my crappy web design can manage.
0:19:43 Um, and I, you know, I don't
have a deadline, right?
0:19:47 I don't have somebody
breathing down my neck.
0:19:48 I don't have some investors
I got to keep happy.
0:19:51 It's like, no, it's like, I'm
going to polish this thing.
0:19:54 It's, and it's a beautiful, feeling
of kind of craftsmanship, which I have
0:19:59 rarely had before in my professional life.
0:20:01 You know, often you feel
like you're a tradesman.
0:20:03 You're just like knocking stuff out
for a customer or for a deadline.
0:20:06 But suddenly I could feel
like, sounds pretentious, but
0:20:09 almost like an artist, right?
0:20:10 You know, artisanal software
that's like perfectly polished
0:20:14 to my, to my specification.
0:20:16 And that was an interesting
journey in its own right.
0:20:18 By the way, it's also a curse because
you could just end up polishing and
0:20:22 polishing and polishing and then
forgetting to actually share it with
0:20:25 you, share it with the rest of the world.
0:20:27 So, you know, there are
downsides to this as well.
0:20:30 But no, I always wanted to invest
in test coverage, and documentation.
0:20:35 And by the way, and this is an interesting
side story we may not have time for,
0:20:39 but like, as a result, you build lots
of tools to help you do that too, right?
0:20:44 So I've now got tools that help me
generate the documentation, tools
0:20:48 that help me generate test cases.
0:20:50 The documentation, as you identified,
has lots of examples and sample code.
0:20:54 Each piece of sample code
is also a unit test, right?
0:20:57 It'll actually take the code
out of the documentation and run
0:20:59 it which is obviously stuff
that I had to go and build myself.
0:21:02 but that was all me, you know,
thinking, well, I don't want to, I
0:21:06 don't want a piece of code in the
documentation that doesn't actually run.
0:21:09 Like, you know, what happens
if I refactor the code and I
0:21:11 forget that it was in the docs?
0:21:13 I know exactly what you're talking about.
0:21:15 There's a little thing that comes to mind.
0:21:17 I'm not sure where I've seen it, but
I think it was along the lines of
0:21:21 like a yuck yuck tree, like the,
the concept of like yuck shaving and
0:21:27 like a tree of dependencies or like
how you would try to do one thing.
0:21:31 And to do that, you had to like
diverge into a different side
0:21:36 project and then your side project
has a side project and you see like
0:21:40 a dependency tree of Yuck Shaving.
0:21:43 So I know exactly what
you're talking about.
0:21:45 That's right.
0:21:46 That's right.
0:21:47 And each one of those tools for
the tools for the tools, each one
0:21:50 of them has to be perfect as well.
0:21:51 Right?
0:21:51 So, anyway, it's probably a fractal thing.
0:21:54 , but yeah, that, that, that itself has
been a really interesting journey trying
0:21:57 to build the, the software that I
really always yearned to be able to,
0:22:03 to build even as it scales, which
is, you know, tricky to be honest.
0:22:07 And obviously once people start using
it too, because then now you've got
0:22:11 opinions coming from people outside
saying, Oh, well, you know, you
0:22:15 might think it's perfect that you do
this, but we don't like it like that.
0:22:18 You're like, ah, now
I've got this dilemma.
0:22:20 Like, am I going to break my
perfect thing to do the thing
0:22:24 that actually a customer wants?
0:22:25 Well, of course, you know,
because if no one's using it,
0:22:28 then then what was the point?
0:22:29 So there is a a whole set of
interesting trade offs that
0:22:33 you then start, start making.
0:22:34 But no, just again, true to its
name, I would say the most interesting
0:22:38 part of that is how did, how do
you make it as small as possible?
0:22:43 and I, maybe that's not so important these
days in the world of, faster internet
0:22:46 connections, et cetera, and good bundlers.
0:22:48 But like, I wanted to see, like, could
I make this like as almost, you know,
0:22:53 infinitely as small as I could make it?
0:22:55 and so I spent a lot of time
thinking about minification,
0:22:57 a lot of time thinking about.
0:22:59 you know, how to refactor the code in
a way that, you know, made it compress
0:23:03 really small and, you know, it's still,
it's still pretty small for what it does.
0:23:06 TinyBase
0:23:06 That is awesome.
0:23:07 I highly encourage everyone who's
watching this to also check out
0:23:11 your beautiful website and the
documentation, the examples, et cetera.
0:23:16 I think you've laid it out very clearly.
0:23:18 but given this is a podcast where
most people listen to, would you
0:23:23 mind in words, trying to describe
what TinyBase is, like, for example,
0:23:29 in the context of a React app.
0:23:31 Okay.
0:23:32 Well, firstly, TinyBase.
0:23:34 org will do a better job of
explaining it than I will do now,
0:23:38 so definitely go, go check that out.
0:23:40 , it's very nice that
you say it's beautiful.
0:23:41 I'm no graphic designer and I
actually think the website
0:23:45 could do with a bit of love.
0:23:46 So if anybody would like to come
along and make it look a little nicer,
0:23:49 they are very welcome to file some.
0:23:51 CSS pull requests on me, I would
be very happy to take those.
0:23:54 But, that all said yeah, look,
the basic idea or the strap line
0:23:59 is that TinyBase is a reactive
data store for local-first apps.
0:24:05 And I'm very careful to say
data store, not database.
0:24:08 I don't want anyone thinking
that it literally is a database.
0:24:11 it's a data store, but it just has
characteristics that are perhaps
0:24:15 familiar to people who have thought
about relational databases in the past.
0:24:19 I also say that it's for local-first
apps because I'm riding on the back
0:24:24 of the cool local-first movement
that's happening right now.
0:24:27 I guess there's no obligation to
only use it on local-first apps.
0:24:31 I think, you know, even a permanently
online app that wants some tabular
0:24:35 data store in the browser would
still be able to use TinyBase.
0:24:41 But yeah, that's the principle.
0:24:43 It's in memory.
0:24:44 So the data is either loaded up from
some local you know, to the browser,
0:24:50 some local storage, or session
storage, or IndexedDB, and then
0:24:55 it's loaded into memory, and then
you build your app on top of it in
0:24:59 a way that is fully reactive to the
changes that happen to that data store.
0:25:04 So, in a way, you have delegated all
of the data provisioning, loading,
0:25:11 synchronization, whatever, to TinyBase,
and you just build your app as a pure
0:25:16 function of the TinyBase contents, and
I will take care of getting all that
0:25:20 data updated in the background for you.
0:25:22 Whether I'm pulling it from the web, or
I'm synchronizing it with CRDTs, or I'm
0:25:27 pulling it out of a local store, or I'm
firing it over PartyKit WebSockets to your
0:25:32 friends you know, I'll do all of that.
0:25:35 All you have to do is build a UI that
hooks into tables or rows or cells or
0:25:41 key value pairs in, in the TinyBase data.
0:25:43 and your app will always be
reacting to, to those changes.
0:25:48 So that's the basic principle.
0:25:49 It doesn't have to use
React, incidentally.
0:25:51 Like, there's nothing to stop you
just, you know, listening to changes
0:25:54 in the data directly, and then
just, I don't know, document dot
0:25:57 writing out stuff if you want to.
0:25:59 But React does work well because,
obviously, that also has a
0:26:02 sort of reactive principle.
0:26:03 And so you can use hooks to attach
your components to any of the
0:26:09 elements of the, the, the store.
0:26:11 And by the way, you can do
that at any granularity.
0:26:13 So you could have a component that was
just watching one cell of one row of one
0:26:17 table, or you could have a component that
was watching a whole set of tables, or
0:26:21 one set of key value pairs, what have you.
0:26:24 Lots of different ways to listen.
0:26:26 And you know, TinyBase will make the,
or do the job of, you know, alerting you
0:26:31 when just that thing that you've looked,
you're looking at has, has changed.
0:26:35 And the same then goes for queries
against the database, by the way.
0:26:38 So if you want to do something more
interesting than just watching raw data,
0:26:42 you also want to look at aggregates
or you know, relationships between
0:26:46 data, or in fact, just queries.
0:26:48 construct arbitrary queries.
0:26:50 You can then attach your app to
listen to the results of those.
0:26:53 And again, I'll take care of all
the reactivity behind the scenes
0:26:57 and your app will update when,
when when those results do.
0:27:01 So I guess when I explain it like that, it
sounds like TinyBase is doing quite a lot.
0:27:05 and that's probably why, you know,
we're two years in and it's taken me
0:27:09 quite a long time to get to this point.
0:27:10 But yeah, that's the basic principle.
0:27:12 And I would say, I think this is
kind of an interesting approach
0:27:17 to apps, whereby You know, how the
data gets into the memory of the app
0:27:22 is kind of, like, delegated away.
0:27:24 And you, as an app builder, now
don't need to worry about things like
0:27:29 going on or offline, or things like
synchronizing, or things like storing.
0:27:33 Like, you should expect your data
storage layer to kind of handle
0:27:37 that for you, and you are then just
painting your app out of the palette
0:27:42 of data that is provided to you.
0:27:46 because As I'm sure you know, and
I'm sure many of the listeners know,
0:27:50 the acts of synchronization and
storage and dealing with all these
0:27:54 network conditions is just horrible.
0:27:56 And rather a small number of people
deal with those problems than have
0:28:00 everybody trying to build a local-first
app having to re solve those problems
0:28:03 is probably the long way to go.
0:28:06 and so I, I, I guess that's
what I'm hoping happens.
0:28:09 And it's not just TinyBase, of course,
there are lots of other people in this
0:28:11 space, yourself included, but, you
know, many other vendors at this point
0:28:14 who are trying to crack these problems.
0:28:16 and I think it's because it's hard
and we know that we have to get
0:28:19 this layer of infrastructure going
before the, oh the local-first
0:28:26 ecosystem, you know, really takes off.
0:28:28 I suspect that at the moment we
may have more vendors building
0:28:32 solutions than we actually have apps.
0:28:34 That's.
0:28:35 Bit of an offhand assessment there,
but, you know, there are a lot of
0:28:38 people trying to solve these problems.
0:28:40 And I think it's because
it's genuinely hard.
0:28:42 And what I really hope is that
we can make it easy enough
0:28:45 collectively that that people can
build these apps much more simply.
0:28:49 And you get the benefits of local-first
without having to have a degree in
0:28:52 computer science for synchronization.
0:28:54 That's what we're all trying to get.
0:28:56 Totally.
0:28:56 And, and I think we're getting really
close to, to that where I think right
0:29:01 now the de facto, solution for state
management on the client might rather look
0:29:07 something like MobX or some people might
still like use Redux or Redux toolkit,
0:29:12 et cetera, or, or like React query.
0:29:15 and I think it's hard enough to build
an app that works in the right way.
0:29:20 And most developers don't, dare
yet to think about persistence
0:29:25 and working offline, trying to
make it work, just like that.
0:29:29 And I think a technology like TinyBase
can really, raise the, the floor quite a
0:29:35 lot in terms of what a state management
solution can give you out of the box.
0:29:39 It gives you all of the things something
like MobX does, but also on top of it
0:29:44 gives you persistence, gives you offline
functionality, gives you syncing.
0:29:49 So I think we just,
yeah, raised the floor.
0:29:53 So in terms of the different
responsibilities, I'm curious to hear
0:29:57 a little bit more, and I think what
stands out about TinyBase here is is
0:30:01 that you really went above and beyond
to make things flexible and pluggable.
0:30:05 So for the persistence path, I think
you do allow a whole different range of
0:30:11 different options to persist, whether
it's IndexedDB or, or other options
0:30:15 as well as for, for the syncing.
0:30:17 So would you mind sharing a little
bit more about the details and
0:30:22 TinyBase syncing evolution
0:30:22 Yeah, certainly.
0:30:23 So in the very first instance, I
was just jSON serializing the content
0:30:29 of the store and writing it to local
storage, or to session storage, or to
0:30:34 file, because you can obviously run
TinyBase on a node server as well.
0:30:38 And, that's great, and in 2022 that
seemed like the Basically, you know,
0:30:45 when you're thinking about small stores
anyway that seems like a fine array of
0:30:49 ways to do it, but Obviously people start
to push the boundaries of what you can
0:30:53 store in those media and just serialize
JSON isn't necessarily the way to go
0:30:56 So I think IndexedDB was probably the
next one that I tackled was not as easy.
0:31:00 , just Just as a small side
note, IndexedDB is not the most
0:31:05 enjoyable API to work with.
0:31:08 Most notably, it lacks any form of
reactivity whatsoever that I know of.
0:31:12 I think there were some proposals
that never went anywhere.
0:31:15 But there's no observability.
0:31:17 So it's kind of Crappy to have
to poll for changes, but , yeah,
0:31:22 I do now have that as well.
0:31:24 So you can either save your in
memory store to IndexedDB or load
0:31:29 it or automatically save to it when
changes happen or automatically load
0:31:33 from it with a polling mechanism.
0:31:35 so that, that, that, that's there as
well and and then around the same
0:31:40 time, in fact, it was Matt Wonlow
that I know it was on your show a
0:31:43 little while back he was working on his
CR SQLite solution and I started to see
0:31:50 that SQLite was starting to emerge.
0:31:53 as as a browser opportunity.
0:31:57 And so I figured, well, you know, there'll
be people who want to store it in the
0:32:00 browser on, on a SQLite database as well.
0:32:03 So I went down the path
of supporting that.
0:32:06 So the tables that were in your
memory store in TinyBase can
0:32:10 now be literally written out
to tables in a SQLite instance.
0:32:14 Or read in, and again,
reactivity is not perfect.
0:32:19 , some APIs are better than others,
but you know, do what I can to, to
0:32:23 keep that as synchronized as possible.
0:32:26 And that then opened the door
to support things like PowerSync
0:32:31 and, uh Turso and ElectricSQL.
0:32:37 , I also did.
0:32:38 PartyKit along the way, which is
not a SQLite store, but it's a
0:32:42 kind of a WebSocket based way of
storing data up in a durable object,
0:32:46 and so it supported that also.
0:32:49 So yeah, there was a period, probably
over the last year or so, that the
0:32:52 version 4 series of TinyBase, if you like,
where each major release was me adding
0:32:56 support for one of these new storage
technologies or synchronization platforms.
0:33:02 The other two that are very worthy of
note are AutoMerge and YJS, so those
0:33:08 are more classic CRDT technologies,
but you can store TinyBase into a
0:33:13 Yjs document, or load it from a Yjs
document, or into an automerge document.
0:33:18 And then, of course, you've
got the benefit of those two
0:33:20 platforms synchronizing in
whatever way they want to.
0:33:23 So, yeah, I guess you could say
I, am providing a lot of options.
0:33:28 Maybe too many.
0:33:29 I, Suspect that when you go to
the website and you see all these
0:33:31 baffling options, you're like, Oh
yeah, but which do I actually choose?
0:33:34 so what I'm hoping to do actually fairly
soon is write a guide for, you know,
0:33:38 this is the kind of app I'm building.
0:33:39 This is the kind of platform I should
use, or this is how I should synchronize.
0:33:43 Do I do the synchronization myself or
do I rely on some third party to do it?
0:33:47 And at the same time, obviously we've
seen companies like PowerSync and
0:33:51 ElectricSQL, you know coming forward
with their, their solutions here.
0:33:54 And so, yeah, if people are using
TinyBase and they want to Provide
0:33:58 a a gateway to those systems,
then that's that option as well.
0:34:01 So yeah, I have tried to
be as flexible as possible.
0:34:04 One of the privileges, I guess, of
working on this as a hobby is that,
0:34:07 you know, I can be open to working with
as many of these partners as possible,
0:34:12 whether they're commercial startups
or just open source projects, right?
0:34:15 I'm not in competition with anybody.
0:34:18 And so if I'm providing a way for
developers to onboard onto these
0:34:21 different things, then that's great.
0:34:22 I definitely have a view that
the, the tide is rising up.
0:34:26 I'm saying that on a boat but the local
tide is rising, local-first tide is
0:34:30 rising, and I'm happy to see all boats
go with it even in the long term.
0:34:35 , even if in the long term TinyBase
becomes not needed because, you
0:34:40 know, all of these other platforms
have, have matured to the point where
0:34:43 they're offering what I already do.
0:34:44 But I think in the meantime, I'm hopefully
providing a nice, disambiguation
0:34:49 layer or you know, something like
that to help people decide what their
0:34:53 approach is going to be without having
to rewrite their app dramatically.
0:34:57 So yeah, today you could store
your TinyBase data to local
0:35:00 storage and tomorrow you could
then sync it to ElectricSQL.
0:35:03 Why not, right?
0:35:04 So in terms of the data that is persisted,
let's say to IndexedDB or OPFS, SQLite,
0:35:11 or synced to one of the syncing providers,
how do you bring the data into memory?
0:35:18 And do you typically, let's say
I have a, let's say I'm using
0:35:22 Turso and have my SQLite database
somewhere, Is TinyBase automatically,
0:35:27 hydrating all of that data from the
database into the in memory version?
0:35:32 Or is there some sort of scoped
version where maybe the database is two
0:35:36 gigabytes and you only can constrain
yourself to 200 megabytes in memory?
0:35:42 how are you handling that?
0:35:43 Right.
0:35:44 So for all of the database based
persisters, I call them, there's a
0:35:48 configuration where you say which
tables you want and, you know,
0:35:52 whether you want to just load from
those tables or whether you also
0:35:55 want to be able to save back to them.
0:35:57 the worst thing I could think of would
be somebody with some large production
0:36:00 database and they connect TinyBase to
it and the next thing you know, it's
0:36:03 tried to load the entire database into
memory or it's tried to write No, back
0:36:08 to the whole database, and then, you
know, that's the end of the world.
0:36:10 So no, it's all very configurable
on a table by table basis.
0:36:15 And the one thing I have not done,
but I'm pretty sure I need to
0:36:18 do, is also provide some kind of
pagination or filtering on that view.
0:36:23 because if you've got a database that
has, let's say you, you, you shard
0:36:27 it on user or something, and you only
want to be able to see just the data
0:36:31 from that user, which, you know, it's
gonna be a pretty common use case,
0:36:34 then make sure that persistence is
not for the whole table, but just
0:36:38 for the relevant rows, of that table.
0:36:41 I think it's going to be
harder than I think it is.
0:36:43 I suspect there are lots
of gotchas to doing that.
0:36:46 , and so for now, there's my, my solution
or my suggestion will be that people
0:36:51 doing this should probably have a per
user database, which is kind of an
0:36:56 interesting approach that I know some
people have started taking anyway,
0:36:59 for, for edge based, databases.
0:37:02 Um, so that, you know, you're,
you're in no danger of, accidentally
0:37:06 loading data from a different
user into someone's browser.
0:37:09 But no, that's definitely
something I need to tackle.
0:37:12 So it's scoped, to answer your question,
it's scoped to individual tables as
0:37:16 to whether it's load or save or both.
0:37:18 But I think over time we
need to add more pagination.
0:37:21 So that you are just looking at,
you know, the top 10 records,
0:37:23 100 records, or what have you.
0:37:25 The reactivity gets really tricky
at that point because I'm sure you
0:37:29 know, you know, SQLite's reactivity
out of the box is pretty weak.
0:37:32 At best, you're going to
know that a table changed.
0:37:35 And what are you going to do?
0:37:36 Query the whole table to find out
which row it was that changed.
0:37:40 And so knowing that it was row 47 out of
2 billion that was the one that changed.
0:37:45 is not currently something that the
database platforms provide by default.
0:37:49 Um, So, that's one thing where I
am really having to work around
0:37:54 those limitations right now.
0:37:56 , and I'm hoping, if I can't make it
clearer to you, I'm hoping that people
0:37:59 working on SQLite are going to crack
this and provide, you know technologies
0:38:03 that make it easier to synchronize
at a much more granular level, or get
0:38:06 reactivity at a much more granular level,
so that TinyBase isn't having to poll
0:38:10 everything in a way that doesn't scale.
0:38:13 Yeah, I mean, I, I think, Ideally
for SQLite, the most principled and
0:38:21 foundational approach would be to
rebuild SQLite from the ground up to
0:38:27 be incremental and to be reactive.
0:38:29 And I think actually a former coworker
of yours, who is um, Julian, who
0:38:35 has also created, like HipHopVM
at, at Facebook and, and HackLang.
0:38:40 he's actually trying to, to do that with
a new startup called, Skip that is a
0:38:45 very ambitious, or even audacious goal to
build the whole new database from scratch
0:38:50 to be reactive, from the ground up.
0:38:53 Which I think is very, very interesting,
but short of that, I think you, you're
0:38:57 left by imposing, reactivity, primitives
on top of SQLite and the best you can
0:39:04 do there is to do the minimal amount
of work, you need at a given situation.
0:39:09 So instead of pulling the entire table
that, you know, okay, row 47 has changed.
0:39:16 And that requires quite a bit of like
a reactivity system on top of it.
0:39:20 And so luckily with, a signal like
system, that is a, it's a pretty
0:39:25 good, primitive to put on top of
it, to implement that reactivity.
0:39:30 But, yeah, it's not as efficient
as you, if you'd build it
0:39:33 from the, from the ground up.
0:39:35 That being said, SQLite is so fast that
you get a lot of performance benefits out
0:39:40 of the box and you can get away with it.
0:39:42 Yeah, that's true.
0:39:43 you know, certainly on the client.
0:39:44 Um, yeah, I, I, without, you know,
sharing too much, internal stuff,
0:39:49 there was a big push, at Facebook.
0:39:51 I'm thinking six, seven years ago, you
know, to think about reactivity for
0:39:56 the entire Facebook stack, because this
concept of, you know, taking the ideas
0:40:00 of React that were out on facebook.
0:40:02 com and bringing them all the way back
through the web servers, which are
0:40:06 HHVM, all the way through to the data
stores that were being used at the
0:40:10 company, you know, like it's a utopian
vision, but like, if you can make
0:40:14 that push based all the way through.
0:40:16 That's spectacular.
0:40:17 and you know, Julian and I were
probably in many of the same, uh
0:40:22 discussion groups and, you know,
internal Facebook groups where a lot of
0:40:25 these ideas were being bounced around.
0:40:27 And, and so yeah, when he left and
went off and did, Skip and you know,
0:40:31 I guess maybe some of this informed my
thinking about TinyBase too, you know,
0:40:36 that dream is still very much alive.
0:40:40 and yeah, it's coming from, ironically,
it's like it's coming from React.
0:40:44 That idea is coming from React.
0:40:45 It's just like, well, okay, can we bring
that idea a little bit further back,
0:40:48 and then a little bit further back,
and then a little bit further back?
0:40:51 Oh, like, and then we
end up on the database.
0:40:53 Like, why can't we get the database to be,
you know, telling us about these changes?
0:40:57 And, yeah, I guess that
would be the vision.
0:41:00 Unfortunately, I haven't checked in
recently, but I think SkipDB wasn't
0:41:04 Wasn't at a level where I could start
working with it, when I last looked.
0:41:08 Um, so yeah.
0:41:10 I think it's not yet fully production
ready, but as far as I know,
0:41:13 they're just still working on it.
0:41:15 So in terms of the query layer
for TinyBase, you have, not
0:41:21 implemented SQL as a query language
on top of your own data store.
0:41:25 Um, so yeah.
0:41:25 But you've provided, given that
JavaScript or TypeScript is the
0:41:29 primary way how you interact with
it, you've just embedded a DSL, as a
0:41:34 query language directly into TinyBase.
0:41:36 So would you mind briefly describing
that and, how that describes a user
0:41:40 experience or the developer experience?
0:41:42 Right, right.
0:41:43 So the, the journey here for me was
that first I knew there were going
0:41:47 to be some very simple query like
primitives that I wanted to have.
0:41:51 So one is, like a metric, which
is some kind of, you know, number
0:41:57 derived from the content in a table.
0:41:59 And I knew that that was going to be the
first thing I wanted to have if I was
0:42:02 building this hypothetical GitHub app.
0:42:04 I was going to want to have a number
in the top that said number of repos,
0:42:07 and I was going to want to have a
number that said number of commits.
0:42:10 And so I actually baked in.
0:42:12 a kind of a metrics oriented DSL where you
can define a metric which is, you know,
0:42:19 basic aggregates, counts, averages, sums,
those sorts of things, which by default
0:42:24 is normally just counting the rows in a
table, but that is then a reactive entity.
0:42:29 So again, you can say, I just want
this span in the top right hand
0:42:33 corner of the window to just always
show whatever that metric was,
0:42:36 and then you can forget about it.
0:42:38 And if that table is updated, You
know that number's going to change,
0:42:40 TinyBase will take care of it.
0:42:42 another thing that I knew I
was going to need to do was
0:42:44 relationships between tables.
0:42:45 So I built a very specific thing for
just one local table, one remote foreign
0:42:51 table, and then like keys between them.
0:42:53 So the value of one column in one table
is used as the identifier for another.
0:42:57 And so, yeah, you want to see the issues
just for this repo, or you want to see
0:43:01 the repos just for this user, then you'll
be able to do those kinds of things.
0:43:04 So those are, were actually baked in
pretty early to TinyBase, just because
0:43:07 they seemed Very important, but I, I
guess I knew eventually people were
0:43:13 going to want to push that on and start
doing things like arbitrary queries.
0:43:20 And I toyed with the idea of, you
know, supporting SQL in some form.
0:43:26 And I should say, I love SQL.
0:43:28 I have been writing SQL most of
my professional career, which
0:43:30 you know, goes back a fair way.
0:43:33 And my most recent role at Facebook
was, you know, the data analytics
0:43:37 team in the data infrastructure org.
0:43:40 So you know, we were working
with SQL all the time.
0:43:43 Perhaps despite that, or because of
that, I kind of got nervous about doing
0:43:47 an arbitrary SQL parser, evaluator,
executor, and then being able to make
0:43:53 the results reactive, which was like,
that's the, that's That's the non deniable
0:43:58 requirement, like, it has to be, reactive.
0:44:01 So, I didn't really have a lot of
choice, but, oh, and by the way, you
0:44:06 know, a full SQL parser with all the
dialects you'd want is, like, it's
0:44:09 gonna quadruple the size of TinyBase.
0:44:12 It's just, it's code based.
0:44:13 So, no longer tiny.
0:44:15 Yes.
0:44:15 So, I wanted to come at it from
a slightly different direction.
0:44:19 I know I'm not the only person to have
thought of doing more like a DSL approach
0:44:22 to this, but I, I wanted to see whether
I could capture as much of the, the
0:44:26 valuable parts of SQL without turning
it into a place where people could build
0:44:31 stupid cartesian joins and kill their app.
0:44:34 so I wanted to make it kind
of a little bit on rails.
0:44:37 And yeah, it was the idea of TinyQL was
born, which is kind of a stupid name.
0:44:41 But it's, as you said, a DSL
that allows you to, you know,
0:44:45 select, where, group, limit.
0:44:48 , actually, no, I don't
have a limit in there.
0:44:50 I limit a later part in the pipeline.
0:44:53 But, yeah, basically allows
you to join tables together.
0:44:56 and then where them, there's havings and
groups and those kinds of things too,
0:45:02 which like really is what 98 percent
of people want to do with SQL anyway.
0:45:07 And haven't really come across too
many queries that I can't express
0:45:11 with these five or six quote keywords.
0:45:14 Yeah, the cool thing about
SQL is you can do anything.
0:45:17 The awful thing about SQL
is you can do anything.
0:45:19 And, I have worked with enough people
who thought they knew what SQL did and
0:45:24 didn't quite know what it did to, you
know, see the trouble you can get into.
0:45:27 So I really, yeah, wanted to make it a
little bit more constrained than that.
0:45:31 And it's, it's been,
it was a real journey.
0:45:33 That was a very tricky, for me, anyway.
0:45:35 , that was a very tricky thing to
build, especially making sure
0:45:38 that all the results are reactive.
0:45:39 Because, by the way, you know,
any of those tables could change.
0:45:43 Any of the rows in those
tables could change.
0:45:44 In fact, you don't want to react
to the changes of the results if
0:45:47 some of the irrelevant rows change.
0:45:49 You only want to know if it was
the relevant rows that changed.
0:45:51 So you have to set up listeners on
all the things that were relevant and
0:45:54 not the things that were irrelevant.
0:45:56 so that ended up being a little
harder than I thought, but yeah,
0:45:59 pretty proud with how it turned out.
0:46:00 And as a result, and I would urge
people to go check this out, You can
0:46:04 build kind of interesting analytics
apps, right, with this, in the browser.
0:46:09 So if you go to the TinyBase.
0:46:11 org demos, you'll see there's a demo.
0:46:14 I think it's called car
analytics or something like that.
0:46:17 And it takes a data set of car
data, different models, different
0:46:20 makes, miles per gallon, years.
0:46:23 So there's a bunch of numerical measures
and there's a bunch of dimensions.
0:46:27 And in the browser, it loads
all the data in pretty quick.
0:46:30 It's not that much data, but you
know, loads it in pretty quickly.
0:46:32 And then you can do groups,
sorts, filters, across all
0:46:35 of this data in the browser.
0:46:37 And it's like, it's completely instant.
0:46:39 It's like, great, love it.
0:46:41 and look, this is not going to be
great for querying the Facebook data
0:46:44 warehouse, however many hundreds of
petabytes that is these days, but
0:46:49 it is going to be fine if you've got
10 or 100, 000 rows in your browser.
0:46:53 And Don't be scared, people.
0:46:55 Like, getting a hundred thousand rows
of data into a browser is not hard.
0:46:58 The browser is perfectly
capable of doing that.
0:47:00 You know, most even phones
are very happy to run that kind
0:47:04 of sized data in, in memory.
0:47:06 so yeah, for a lot of even analytics
use cases, you can do, you could, you
0:47:09 could build interactive dashboards
with this kind of technology and
0:47:13 have a sort of constrained set of
queries all running off TinyBase.
0:47:17 And it's super feasible, and it's been.
0:47:20 It's been fun to build that.
0:47:22 Whether or not people are using
it in the wild, I don't know.
0:47:24 I should say, I don't put any
instrumentation into this product.
0:47:27 So who knows what people are using
it for, or how they're using it.
0:47:31 But I hope that those people who use
it do, do find the value of being
0:47:34 able to run queries, without, shooting
themselves in the foot with SQL.
0:47:37 Did you follow any sort of prior
art in regards to how you built
0:47:41 that reactive query system?
0:47:43 So, as I was working on a similar
part as, the work on Rffle, there,
0:47:48 there was, some prior art in
regards to Adapton and mini Adapton.
0:47:53 And so this was, some inspiration,
that we followed for the reactivity
0:47:57 system and we've like over time learned
that, it's very similar to how now like
0:48:04 signals is all the rage these days.
0:48:06 It's like very similar
in terms of the ideas.
0:48:09 So did you follow some,
some similar paths?
0:48:12 No.
0:48:13 And I wish I had.
0:48:14 I think it's one of my weaknesses.
0:48:16 One of my weaknesses is
that I look at a problem.
0:48:18 You know, it's like the
famous Hacker News thing.
0:48:19 It's like, I could have
built that in a weekend.
0:48:22 I, I look at a reactive query system.
0:48:23 I think, yeah, I'm sure
I could build that.
0:48:25 And then like two months later, I
was like, I wish I'd read a bit more.
0:48:30 I am not good at prior art and
reading scientific papers or even
0:48:35 just, you know, looking at other
projects to see how they're done.
0:48:37 I think, you know, I know what
the result should be and I'm
0:48:39 going to try to build it myself.
0:48:40 And then I get into trouble.
0:48:42 Fortunately, I worked
my way out of trouble.
0:48:44 If it makes you feel any better,
even with knowledge of the prior art,
0:48:48 it still took easily more than two
months just to get anything working.
0:48:53 So,
0:48:55 Okay, that's just, that's just my,
my weakness that fortunately I
0:48:59 was able to work around this time.
0:49:00 But yeah, I think the other thing is
that I obviously knew in intense detail
0:49:06 how the reactivity of the underlying
tables, rows, and cells were right.
0:49:11 And so I knew how I was gonna have to
cascade the listener trees into all
0:49:16 the relevant parts of the database.
0:49:18 And, That was, yeah, that was just
something I had to figure out for myself.
0:49:23 The other thing I alluded to
earlier is that TinyBase is
0:49:27 just draped with test cases.
0:49:29 Like, I am exhaustive, overly exhaustive.
0:49:32 I'm completely OCD when
it comes to test cases.
0:49:35 It's not quite TDD.
0:49:37 I don't always do the tests first,
but, you know, I do eventually try
0:49:41 to make it look like I did TDD.
0:49:42 So I built every possible query I can
think of, and I build all these demo
0:49:46 apps, and I build all these sample,
you know, fragments of code, and
0:49:50 they all get tested exhaustively, and
so I figure out, well, once, once,
0:49:55 once those work, I'm probably there.
0:49:57 So that, that was my
benchmark, and I got there.
0:49:59 It's probably, I'm sure it's
still possible to break it,
0:50:02 but, you know, I think, I think
I'm about where it needs to be.
0:50:06 Schema migrations
0:50:06 So one of my favorite problems
to, to think about, and it's a
0:50:10 love hate relationship in terms of
technical problems, which are schema
0:50:14 migrations and schema evolution,
which is something I thought a lot
0:50:18 about, through the work on Prisma.
0:50:20 And I tried for the work on Overtone,
ideally having to deal with as little
0:50:27 as possible, or at least have a
very principled foundation for that.
0:50:30 And I feel this is a bit of
a underdeveloped area in some
0:50:34 of the available open source
technologies right now, or
0:50:36 local-first technologies right now.
0:50:38 So I'm curious how you have been thinking
about schema migrations for TinyBase.
0:50:45 and if you've already built
some foundation there.
0:50:47 Okay.
0:50:48 Well, before we talk about schema
migrations, we should talk about schemas.
0:50:51 So.
0:50:52 even having a schema was kind of a
late thought for me, TinyBase outta the
0:50:57 box by default, when, and especially
when I first built, it, didn't have
0:51:00 the concept of a schema at all.
0:51:03 And, you know, any row
could have any set of cells.
0:51:07 I mean, a, a a row of cells
was basically just a map.
0:51:10 So it was possible for every row of
a table to have completely different
0:51:14 cells in it, which is a bit.
0:51:16 Scary, and contrarian especially
for someone like me who'd
0:51:21 just come from working on the
world's largest data warehouse.
0:51:24 the idea of doing this
was, was a little daunting.
0:51:26 But,, I felt like, you know, the word tiny
meant I was allowed to do things like that
0:51:32 and take away some constraints and just
let people put arbitrary data into this
0:51:38 basically a map of map of maps, right?
0:51:39 Tables of rows of cells.
0:51:41 Now that obviously is fun for little
demo apps, but at some point you
0:51:46 need to start tidying things up.
0:51:48 It's all very good when it's just
one person building it too, right?
0:51:50 Because I can remember all the fields
that are supposed to be in this table.
0:51:53 but when I've got colleagues who
maybe want to know what the schema is,
0:51:55 then should do something different.
0:51:57 So I did, yeah, we do have schema support.
0:51:59 I did add schema support,
but it's optional, right?
0:52:02 You, you, you have to explicitly
add the schema onto your store.
0:52:06 but even then the schema is,
I'm afraid, sorry, very basic.
0:52:10 So you can say whether a cell is a, you
know, string, a boolean or a number.
0:52:13 And you can say whether it was required
or not, and what the default is.
0:52:17 That's it.
0:52:18 So that seems a little daunting.
0:52:20 I should say, by the way, that TinyBase
only supports those data types anyway.
0:52:24 So you can't put anything other
than a string, a number, and
0:52:27 a Boolean into a field anyway.
0:52:29 So I don't have support for
arrays, or objects, or even null.
0:52:35 so that kind of makes you realize this is
a relatively simple thing to model anyway.
0:52:41 But yeah, that, that, that's, that's
where we are right now with schemas.
0:52:46 And once you've added a schema,
it will try to massage the data
0:52:51 into that schema, and it'll drop
anything that doesn't match.
0:52:54 that's pretty basic, I will admit.
0:52:55 But there are two, I think, two main
things I need to do from here with this,
0:53:00 because clearly that's not sufficient.
0:53:01 It's great for fun, simple apps.
0:53:03 It's actually, I will say, it does allow
you to do more than you might think.
0:53:07 But anyway, people do want to do arrays,
and That lets you do many to many joins,
0:53:12 and they do want to do objects, so fine.
0:53:14 We should probably get
to that at some point.
0:53:16 So, that's, yeah, that's my big push,
I think, is making time based support
0:53:20 richer data types inside a cell.
0:53:22 I think arrays will be the first.
0:53:24 I'll probably stick to having arrays
of one, type, like it's going to
0:53:27 be all numbers or all strings.
0:53:29 I don't think I'm going
to have mixed arrays.
0:53:31 That could get too tricky.
0:53:32 I still want to have
some kind of validation.
0:53:34 so we'll do that.
0:53:35 , that knocks off a many to many requirement
that a bunch of people have had on GitHub
0:53:39 that I can't currently do very easily.
0:53:41 And so we'll, we'll do that.
0:53:43 Secondly, I would love to do
more with the schemas themselves.
0:53:48 So the, schema dialect that I put
in right now is just, like I said,
0:53:53 a fairly simple description of what
the rows are, but There's Zod in the
0:53:57 world, and there's Effect, and a bunch
of other ways that people like to
0:54:01 express schemas, and if they've already
built their schema for some other part
0:54:05 of the app, or they've built it for
some, you know, some other part of the
0:54:08 system, maybe up on the server, like,
how can they then turn that into the
0:54:13 schema that resides in the app itself?
0:54:16 So I have no doubt that I will be building
Zod to TinyBase Converter, and Effect
0:54:22 to TinyBase Converter or some, some,
something to map, map those schemas in.
0:54:27 Still doesn't solve your problem
of migration, and I will admit
0:54:30 that I am a bit daunted by that.
0:54:34 In general, I am daunted by the
whole space of anything that
0:54:36 requires build time code mods and,
you know, running generated code.
0:54:42 Like, I just hate that because immediately
you've got a team of 10 people and all
0:54:46 of their artifacts get out of sync.
0:54:47 I've seen that at Facebook, right?
0:54:49 Whether it's GraphQL or what have you.
0:54:51 So I, always gonna be trying to look
for solutions that don't require you
0:54:55 to run a build step, to generate some
magic code that you then have to remember
0:55:00 to link in, how to actually avoid that
step and still do schema migration.
0:55:05 is not quite in my head yet.
0:55:07 So I share your sentiment.
0:55:09 These things are underdeveloped,
but I don't have an answer for you.
0:55:12 Right.
0:55:12 I can share my thoughts on it, which is,
kind of like an architectural one, that
0:55:18 has quite a lot of implications, but those
implications are appealing to me, I would
0:55:23 separate the read path from the write path
and have the write path be event driven.
0:55:29 And that allows me to do
sort of event sourcing.
0:55:33 So similar to how Redux works,
but, persisted and distributed
0:55:37 Redux, if you, if you want.
0:55:39 And, then like the way, how you
interpret your Redux events.
0:55:44 you can interpret them however
you want, and you can very
0:55:47 easily change that over time.
0:55:49 So you basically just go through the
entire event source, the entire event
0:55:53 log of events, the entire history,
and your app database that you
0:55:58 read from, that is just, a result.
0:56:01 By going through all the events
and every event, you might come
0:56:06 across like a user signed up event.
0:56:08 , you then translate into a insert
into users with all the values.
0:56:15 And you can change the user table
at any moment, you can, instead of
0:56:19 having one user table, you can have
a customer's table and an employee's
0:56:23 table and just reinterpret the events.
0:56:26 And so this way, you don't
have schema migrations at all.
0:56:29 You just change sort of the
. Interpretation of your event.
0:56:33 it requires a bit of a different
architectural approach and I'm
0:56:36 building LiveStore all around
this idea, which is very different
0:56:40 from, from other approaches.
0:56:42 But it's the most principled approach
that I can currently think of in
0:56:47 terms of schema migrations, where you
don't do like my schema migrations,
0:56:51 basically just I design a new schema.
0:56:53 And I target that and there's
no, like, up or down scripts.
0:56:58 and so that is very appealing, but
it has other downsides as well.
0:57:02 But this is certainly an idea that I,
that I think is worth exploring more.
0:57:07 No, I like that.
0:57:08 And it also shares, , Similarities
with what I'm separately having to do
0:57:12 for the, native CRDT synchronization,
as of TinyBase version 5, which at
0:57:18 the time we're speaking isn't quite
released, but by the time this podcast
0:57:21 is out might have been released.
0:57:22 So, hey everybody, go download version 5.
0:57:25 but no, it's, it's got a native CRDT
support and I do that by sending,
0:57:29 you know, basically little, little
messages, but those are the data.
0:57:33 level, right?
0:57:33 It's, you know, here's, here's how
this cell changed, and I wonder whether
0:57:37 like there's some higher abstraction
for those messages that is then sort
0:57:41 of schema agnostic or that can be
mapped into whatever the schema is.
0:57:45 Because by the way, the minute you
start synchronizing things, the schemas
0:57:49 getting out of date is going to become
an issue very quickly, because maybe
0:57:53 you've updated the app on your phone,
but you haven't updated the app on
0:57:55 your desktop, but now they're jogging.
0:57:58 You know, WebSockets to each other
and all hell breaks loose because
0:58:01 the schemas weren't quite the same.
0:58:02 It's all questions at the moment.
0:58:04 Yeah.
0:58:04 I think there's a whole episode
just to be done around that.
0:58:07 Happy to form a little working group
around this, but I think that goes
0:58:11 beyond the scope of, of this podcast.
0:58:14 but yeah, I think you've come really,
really far with TinyBase and this
0:58:18 is, I think, a very common area of
the local-first technologies part
0:58:24 that is not yet as fully explored.
0:58:28 and I've just changed my priorities
to go double down on that from
0:58:32 the beginning, since I know like
how gnarly this is to deal with.
0:58:37 Well, can I just say that, you know,
from the point of view of exhaustiveness,
0:58:41 there are many, many unsolved
problems in the local-first space.
0:58:46 And I have a horrible feeling
that it's easy to think that
0:58:50 synchronization is the problem.
0:58:52 And then once you've solved
synchronization, you're good.
0:58:55 Well, I don't think that's true.
0:58:57 And the reason I think that is that
I've tried building some local-first
0:59:03 apps that, you know, have things like
anonymous versus authenticated access,
0:59:08 or that have private data versus shared
data, or that have online, offline, and,
0:59:14 you basically multiply this state machine
together and you've got like, you know,
0:59:18 two to the power of four combinations
of state that your clients can be in.
0:59:23 And.
0:59:23 It's all very well being able to
synchronize data back and forwards, but,
0:59:26 like, you don't want to synchronize a
non shared store with another user, or
0:59:29 you don't want to share an authenticated,
you know, you brought a copy of an
0:59:34 authenticated database to be local,
but now you log out and now that's
0:59:38 got to go, like, the state machine
of what it actually takes to build a
0:59:40 real app like Figma or I don't know.
0:59:45 It's like, wow, there are lots of crazy,
crazy transitions that can change here.
0:59:49 And I don't think anyone's really
thinking about all of those.
0:59:52 I think we all got hung up on, oh,
once we crack CRDTs, we'll be good.
0:59:56 No, no, no, no, that's the easy part.
0:59:57 It's not state management, it's the
state of the state management in a way.
1:00:00 It's like, should I be
synchronizing this data?
1:00:03 Because this person isn't logged in,
or is logged in, or is online, or is
1:00:07 offline, or has shared this document,
or hasn't shared this document, and
1:00:11 that is actually unsolved, I think.
1:00:14 So I built a little sample app called
Tiny Rooms a way back, and like,
1:00:19 the app itself is stupid, it's just
dragging rectangles around, but like,
1:00:21 I tried to model all of these different
states, and it was really hard.
1:00:25 Like, I was, oh god.
1:00:27 I would hate to be actually
trying to build a production grade
1:00:30 local-first app at the moment because
those problems are really hard.
1:00:33 Yeah, I think the, the benefits
are that if you're building linear
1:00:39 or if you're building Figma or
if you're building, Tiny rooms.
1:00:43 the advantage for you is that you have
very specific trade offs, very specific
1:00:49 constraints that work for your app.
1:00:52 And so you can take tons of
shortcuts that for a more
1:00:55 general solution are not viable.
1:00:57 But for your app works.
1:01:00 And so, given that, well, there,
there's many trade offs that
1:01:04 Figma can make that would not work
for Linear and, and vice versa.
1:01:08 But I think that is where you can
still build things in a local-first way,
1:01:12 that is like competitively, beneficial.
1:01:15 but I agree having the best of
all worlds, is where we're a few
1:01:20 days away from that, at least.
1:01:22 Yeah.
1:01:23 So I think that's one of the
curses of being a, a tools
1:01:25 builder like, like you, right.
1:01:27 you know, I feel that I'm building
shovels for people who are going
1:01:30 to go off digging for gold.
1:01:31 but I have to build a shovel
that'll work in all different
1:01:34 types of terrain to be successful.
1:01:36 Um, and so like I said, you know,
is it going to be online, offline,
1:01:39 auth, not auth, stored locally,
stored remotely shared or not shared?
1:01:43 Like, okay, now 16 combinations,
got to support all 16.
1:01:46 Whereas yes, if you're Figma,
you can say, well, no, you
1:01:48 can't ever use it anonymously.
1:01:49 You've always got to be logged in.
1:01:50 It's like, oh, great.
1:01:50 Okay, fine.
1:01:51 Like, I've just reduced
that problem to eight.
1:01:53 Yeah.
1:01:54 Pick your trade offs very carefully.
1:01:56 And so this is why I'm also,
why I'm working on Overtone
1:01:59 and LiveStore at the same time.
1:02:02 So if your app is sort of like
Overtone esque shaped, then LiveStore
1:02:08 will work really well for you.
1:02:09 If it looks rather like Facebook,
like a social network, or has like
1:02:13 some very different shapes and trade
offs, then it's not as good of a fit.
1:02:18 But I think this is also what's so
nice about having all those different
1:02:22 local-first tool vendors is they have all
their different starting points and all
1:02:26 their different thoughts on trade offs.
1:02:29 And, I think this is where we're
going to have a great shovel
1:02:32 for many different terrains.
1:02:34 Yeah.
1:02:35 I think that's exactly right.
1:02:36 You know, it's a dilemma.
1:02:37 Do you build a tool that
fits one niche perfectly?
1:02:41 And, yeah, of course, that really informs,
and you'll build a, you know, brilliant
1:02:45 tool for the one job you have in mind.
1:02:48 but at the same time, you
may have limited your market.
1:02:51 Now, fortunately, I don't have to think
about those things because, you know, it's
1:02:54 not a commercial, commercial business.
1:02:56 and I'm, but I'm trying to keep as
many People are happy as possible.
1:02:58 And if I'm just listening to the
feedback that's coming in on, on
1:03:01 GitHub and people are saying, Oh, I
want it to go off in this direction.
1:03:04 And, Oh, someone else wants
to go off in that direction.
1:03:05 It's like, okay I'm going
to try to, maybe I'm going to
1:03:10 try to keep everybody happy.
1:03:11 but then the complexity falls onto me.
1:03:13 And that's where being a tools
builder starts getting hard work.
1:03:16 You know, this.
1:03:18 Right.
1:03:18 I mean, for, for what it's worth, I
think I'm trying to actually, write
1:03:23 down the cases where LiveStore is
not a great fit for, and I'm trying
1:03:27 to be like, very explicit about
the trade offs that I'm making.
1:03:31 So for example, LiveStore right now
expects you to load all of your data
1:03:35 into a single SQLite in memory database.
1:03:39 Luckily, SQLite is very efficient, like
based on my benchmarks and testing, much
1:03:44 more efficient than like JSON in memory.
1:03:47 It's much more efficient to keep like
hundreds of thousands of rows in,
1:03:51 in memory SQLite as you keep them in
just JavaScript objects in, in memory.
1:03:57 you need to just, deserialize
the, just a few ones that you
1:04:01 actually want to like currently
keep in memory for, for your query.
1:04:05 And so, but that's a major trade off.
1:04:07 If you have your database, if
that's necessarily five gigabytes.
1:04:11 Then LiveStore is probably
not a great fit for you.
1:04:14 Or if you don't want to follow the more
like event sourced, nature of it, then
1:04:20 it's probably also not a good fit for it.
1:04:21 But I think this is what makes it
more specific for when it's a good
1:04:26 fit for someone, whereas if it's not.
1:04:28 So actually this brings us onto
an interesting thing that I think
1:04:31 this community needs, which is.
1:04:34 kind of a, a community view of what
all these pros and cons are of the
1:04:38 different solutions and what the
decision tree should be if you want
1:04:43 to build an app of a certain type.
1:04:45 You know, am I going to be online
or am I going to be offline?
1:04:47 Am I going to be authed or not authed?
1:04:48 You know, how do I work my way down to
what is going to be the best solution?
1:04:52 Because of course, you talk to
a vendor, they're going to say,
1:04:54 oh, well, we can do everything.
1:04:55 but something that allows people to
know when, you know, when should I use.
1:05:00 In something like the more mature
front end world, people know when
1:05:04 they should be using React versus
Svelte or whatever, I guess.
1:05:07 There are plenty of places you can go
and read up about what the pros and the
1:05:12 cons are of these different solutions.
1:05:14 But right now, if you're looking
at Database Sync Technology A and
1:05:18 Database Sync Technology B, they're
both local-first, apparently, right?
1:05:22 But one's going to be better
for one thing versus another.
1:05:24 Who's going providing that commentary.
1:05:26 And I think we haven't got that critical
mass of users yet to be able to say, well,
1:05:30 I tried it and this didn't scale, but
I went to this one and this one worked.
1:05:33 Someone else saying the opposite.
1:05:34 And then you can like have that debate.
1:05:36 and so.
1:05:38 Actually, this is something I didn't
mention, but, you know, one of, one
1:05:40 of the parts of my local-first journey
was, was putting together a directory
1:05:44 of these solutions that I was finding.
1:05:46 And so I launched that as
the localfirstweb.dev site.
1:05:50 And that was partly me just
trying to enumerate what these
1:05:53 different solutions were.
1:05:55 And obviously a community
is built up around that.
1:05:57 And now we obviously have
the conference as well.
1:05:58 And what I'm really hoping is that we
see more and more people building real
1:06:01 apps, learning What trade offs they're
having to make, and which of these
1:06:05 solutions they're then converging on.
1:06:08 given those and, and I think it's fine
to have lots of different types of tools.
1:06:11 Sometimes you need a pickaxe,
sometimes you need a shovel,
1:06:13 sometimes you need a pan to get
the gold out of the river, right?
1:06:16 But it's not always obvious which
tool you should take until you've
1:06:19 actually gone out into the hills
and started looking for gold.
1:06:22 Sorry to stretch the analogy, maybe
it's because I lived in California
1:06:24 for a while, but you know, I think
we're not at that point yet, right?
1:06:27 We haven't had enough, we've not had
enough miners going up into them,
1:06:30 there hills to go look for the gold.
1:06:32 so we haven't formed opinions about
what the right trade offs are and
1:06:36 TinyHub
1:06:36 We will get there.
1:06:37 So speaking of real apps, it seems like
I've recently nerd sniped on Twitter.
1:06:42 I've recently nerd sniped you into
building another app on top of TinyBase,
1:06:48 which is a local-first GitHub client.
1:06:51 And you've made quite a
lot of progress on that.
1:06:53 Do you, do you mind sharing more?
1:06:55 Sure.
1:06:55 So that was a, that was
a clinical nerd snipe.
1:06:58 That was, that was one of the
finest even though I was the victim.
1:07:01 So as I mentioned, you know, one of my
original ideas for an app that would
1:07:04 use something like TinyBase was an open
source dashboard that would let you see
1:07:08 large numbers of repos and, and, and, and
study them and understand them at scale.
1:07:12 And I just never got
around to building it.
1:07:14 And then of course you said something on
Twitter like, Oh, wouldn't it be awesome
1:07:17 if there was a local-first GitHub client?
1:07:19 I'm like, No!
1:07:20 Damn it!
1:07:21 That's what I was supposed to build!
1:07:22 Now I remember!
1:07:23 So I took that and I figured
that I would maybe go and put
1:07:28 my, my code where my mouth was.
1:07:31 So I started a little repo
called TinyHub, which is in the
1:07:36 same , parent org as TinyBase.
1:07:39 So I guess you have show notes.
1:07:41 There'll be a link in the show notes
and, people can go check that out.
1:07:44 And yes, it's a GitHub client for
running locally in your browser.
1:07:48 And I wanted to try and build it as
quickly as I could, just to hopefully
1:07:52 prove to myself that building
apps with TinyBase would be fun.
1:07:56 And what it does is it, there's
a quick bit of GitHub OAuth right
1:08:01 at the beginning, and it then will
pull repos that are either starred
1:08:05 or that you, have personally, or
any orgs that you're a member of if
1:08:09 you accept those OAuth permissions.
1:08:12 Anyway, long story short, it pulls
the GitHub data via the API down
1:08:16 into a local TinyBase instance.
1:08:19 In fact, a number of instances.
1:08:20 It actually has, there's a TinyBase
instance per repo, because if you
1:08:24 pull React, like, okay, now we've got
20, 000 pull requests or something.
1:08:29 So it will pull data locally,
stores it in local storage.
1:08:35 Yes which seems to be fine, actually.
1:08:38 , just, it's a, the first hundred
records which is the basic
1:08:42 pagination for, for GitHub.
1:08:44 So it's, it's not, you know, a
comprehensive open source dashboard, but
1:08:48 it at least gives you A sample of, you
know, 100 repos from 100 orgs and up to
1:08:53 100 pull requests and issues from each.
1:08:55 And yeah, then just paint a
React UI out of that data.
1:08:58 And if you go check the repo,
you'll see I put a little video
1:09:02 on the front page of the readme.
1:09:04 And of course, the beauty of this
is that now all of this GitHub
1:09:06 data is sitting in your browser.
1:09:09 And, it's in memory, but you can
flip through it at 16 milliseconds
1:09:14 a frame, to your heart's content.
1:09:15 So, you know, you press the, you press
the down cursor and it whizzes down
1:09:19 through the pull requests and it's pffft.
1:09:21 Views them all instantly.
1:09:22 Not a spinner in sight.
1:09:24 It's pretty awesome.
1:09:25 I'm not sure it's a particularly useful
tool at this point because it's read only.
1:09:28 Um, and it obviously has to do some work
to pull the data down at the beginning.
1:09:33 But it certainly, I hope, emphasizes
the fact that if you co locate your
1:09:38 data with the app, which I think is
kind of like what this, this, this
1:09:41 local-first thing should be all
about at some point, it's amazing.
1:09:44 Like, the UX is spectacular.
1:09:46 Um, and you can just paginate through
this stuff at the speed of thought.
1:09:50 Which is really amazing.
1:09:52 I will just add a little side note here,
is my philosophy around local-first, or
1:09:56 my motivation for local-first, apart from
being on a boat, is the user experience.
1:10:01 I'm just tired of looking at
spinners, because I'm waiting for
1:10:03 something to pull down and cache
locally and then get thrown away.
1:10:06 Like, if I can get everything into
the browser as soon as possible,
1:10:09 and then just go through it at 16
milliseconds That's, that's amazing.
1:10:13 So hopefully that has emphasized
that point of local-first.
1:10:18 Yes, it's about data governance.
1:10:20 Yes.
1:10:20 It's about, you know,
owning your own data.
1:10:22 Yes.
1:10:22 It's about you know, all the other
things that are laid out in many, many
1:10:26 other essays other than mine, but like
having a UX like that is really, for me,
1:10:31 what it's all about, that's the focus.
1:10:33 , and yeah, so I'd encourage people
to go check out TinyHub and that's
1:10:37 me building what I think is a
state of the art TinyBase app.
1:10:42 I would still say that to make it fully
functional, making it read write is
1:10:47 a little more tricky because whilst
GitHub has some pretty awesome bulk
1:10:51 read APIs the write APIs are a little
different and making changes to a
1:10:58 local datastore and then going back
online and reconciling that back up
1:11:01 to GitHub is not as easy as it sounds.
1:11:04 I'm sure it's Events are probably the
answer to that, is what you'll say.
1:11:07 Um, but but I'm not there yet.
1:11:09 Yeah, again, to my earlier joke that
we have more tools than, apps at this
1:11:14 point, I really want to just add to the
pool of local-first demos, right, but
1:11:19 add to the number of apps that show what
could be done, and really emphasize what
1:11:23 the benefits are of this whole movement
rather than just building shovels.
1:11:27 I'm actually going to go out and
look for some gold myself as well,
1:11:31 Outro
1:11:31 I think that's a wonderful
place to, to wrap it here.
1:11:35 James, thank you so much for coming on
the show today, sharing all of your wisdom
1:11:40 on TinyBase, the path that led you here.
1:11:43 Thank you so much.
1:11:44 It's been a huge pleasure and I wish
the podcast the greatest success.
1:11:48 I'm so excited that you've set it up
and honored to be a small part of it.
1:11:52 And thank you to everybody
that has made it this far and,
1:11:56 followed the TinyBase journey.
1:11:57 It's been a privilege to work on
this and share it with you all.
1:12:00 Thank you for listening to
the local-first FM podcast.
1:12:03 If you've enjoyed this episode and haven't
done so already, please subscribe and
1:12:06 leave a review wherever you're listening.
1:12:08 Please also share this
episode with others.
1:12:11 Spreading the word about the
podcast is a great way to
1:12:14 support it and to keep it going.
1:12:16 A special thanks again to Rocicorp
and Expo for supporting this podcast.