0:44:40 So you mentioned that you haven't
yet built larger systems with the
0:44:45 event sourcing approach, but I
think you've still done a little
0:44:48 bit of research on what might await
you in the event sourcing world.
0:44:53 So could you outline a little bit
of like the potential concerns
0:44:57 you see on the horizon when
going all in on event sourcing?
0:45:01 Yeah, so I guess the main concern
always is if you're Sending around
0:45:06 this log of events to clients.
0:45:09 And if you're storing this as
your single source of truth, then
0:45:13 storing all these events forever,
it might take up a lot of space.
0:45:15 If you could imagine a text document, if
each text character corresponds to 100
0:45:20 bytes of JSON, then the history of all
the events is going to be a hundred times
0:45:25 bigger than the actual text document.
0:45:26 Even if you've since cleared out the
entire text document, now it's empty.
0:45:29 You still have all this state.
0:45:30 So that's the main challenge is just how
do we store the events efficiently, how
0:45:35 do we maybe compact them, say I don't
need these events anymore, I'm going to
0:45:38 throw them away and replace the state,
while still making that play nicely
0:45:42 with, you know, clients who have been
offline for a month, that sort of thing.
0:45:45 Which sort of mechanisms do
you think will mostly help to
0:45:49 overcome some of those issues?
0:45:51 I'm hoping the main mechanism is just
To give up, basically say text is
0:45:56 very small for any, the main sources
of lots of data in your app are
0:46:01 blobs like images or videos, which
you can put somewhere else anyway.
0:46:05 And then for the actual event describing
the fine grained changes, just store
0:46:08 them all and it's only going to be
a few megabytes per document anyway.
0:46:11 Got it.
0:46:13 Yeah.
0:46:13 And I think on top of that, there's
also the compaction use case.
0:46:17 Now that I have a little bit
more, insight on, on that
0:46:21 approach with building Overtone.
0:46:23 for example, given that everything you
do within Overtone, whether it's playing
0:46:28 a track, whether it's navigating within
the app, whether it's adding a track
0:46:31 to your playlist or follow an artist,
all of those are an event and Adding
0:46:39 a track to a playlist, there you do a
lot less of those than, for example,
0:46:45 in the background, the app auto playing
the next track, which is also an event.
0:46:52 And another kind of event is if the app
tries to authenticate with a music service
0:46:58 such as Spotify to exchange tokens, which
it needs to do at least Once an hour.
0:47:05 So it does so a little bit ahead of time.
0:47:07 So, also when you reload the
app, it needs to do that.
0:47:11 So just by the fact by, the app running
in the background over time, it Racks
0:47:18 up quite a lot of different events.
0:47:21 And I think they're the interesting
part is the nature of the events
0:47:25 and the nature of those events also
allows for different trade offs.
0:47:28 So me putting a track into a
playlist, A, there's going to be
0:47:33 like way fewer events of those.
0:47:35 and it's fine to keep the
entire history of this around.
0:47:38 What's so cool about this also, the fact.
0:47:41 That, I have this event allows me to
trivially implement a feature like that.
0:47:46 I can hover over the track and I see
the information when was it added by
0:47:51 whom was it added to, to the playlist.
0:47:53 It also makes implementing things such
as undo much easier, but the other kind
0:47:59 of events, which might be implicit or
which might just be a lot more, higher
0:48:05 quantity, what I've seen is that, it's
not as crucial to keep those events
0:48:11 around for eternity, but some of those
events are then also made irrelevant by
0:48:17 follow up events of the, the same type.
0:48:20 So for example, if your app has
authenticated and overrides sort of like
0:48:24 an off state into the database, and.
0:48:27 two hours later, it has
already done so 10 more times.
0:48:31 I don't need to keep the entire history
before that, maybe besides auditing
0:48:35 reasons, so I can just at some point
remove the old events, which keeps
0:48:41 an otherwise always growing event
log at a, for this given event type
0:48:47 at a much more like constant size,
which makes it much more feasible.
0:48:52 Another thing that I, started thinking
about is like, what if you have not
0:48:57 just like one event log, but what
if you have multiple event logs?
0:49:01 And what if you have, a
hierarchy of event logs?
0:49:04 This is something that I also want to
think a little bit more about, Let's
0:49:08 say you have a, a tree of, playlists,
like a, a folder of playlists.
0:49:13 So you have a, a playlist.
0:49:15 And that playlist could also, possibly
be a folder of other playlists.
0:49:20 So now what does the event log exist for?
0:49:23 Does it exist for like,
everything in my library?
0:49:26 Does it exist for a broken down to.
0:49:30 only giving information about which
playlists I have, and then I need to
0:49:34 subscribe to another playlist, but
what if that playlist is a folder?
0:49:38 So this hierarchical aspect of
it, I think this will keep me busy
0:49:42 for, for a little bit as well.
0:49:43 Do you have thoughts on those problems?
0:49:46 Yeah, I mean, this, the, what
you're saying is really interesting.
0:49:48 It makes me think of the
problem of ephemeral presence.
0:49:52 So, you know, in Figma, when your
collaborators are moving their
0:49:54 mouse cursors around, you can see
where they're at it every time.
0:49:58 I would imagine Figma is not actually
persisting those mouse movements,
0:50:01 it's just sending them over the usual
channels so that you can see them live,
0:50:04 but then you forget about these events
because they don't matter anymore.
0:50:07 So I wonder if you could maybe do that
for a lot of the events that don't
0:50:11 matter as much, or even in a text editor.
0:50:13 So one thing that's really hard with a
collaborative text editor is you'd like it
0:50:17 so that whenever you press a key, that key
is immediately sent to your collaborators.
0:50:21 But if that actually creates an event
that's persisted in the log, then you have
0:50:24 this issue of, you know, 100 times as much
storage as key presses, but maybe what
0:50:28 you could say is when you press a key,
that's like an ephemeral presence message.
0:50:32 It's not actually stored,
it's just sent over the same
0:50:34 channel as the mouse movements.
0:50:36 And this is sort of like an ephemeral mini
log that's stacked on top of the actual
0:50:40 event log, and then every 10 seconds
or so you send a compacted version of
0:50:44 like the entire sentence that the person
typed as a single event, and that's
0:50:47 what's actually stored on the backend.
0:50:49 I wonder if that could help at all, or
if this is even possible to implement.
0:50:52 Right.
0:50:53 I've actually implemented a
small version of that already,
0:50:57 which I call local only events.
0:50:59 The idea of that is that, there's kind
of like hierarchies of syncing as well.
0:51:05 There's like syncing, just from the main
thread to the workers thread, which is
0:51:11 responsible for persisting the data,
but also from one tab to another tab.
0:51:18 And, those two tabs should in
some regards, Converge, and in
0:51:23 some regards, allow divergence.
0:51:25 so for example, if you have Notion open in
two tabs, you want to be able to navigate
0:51:32 to different documents and those different
tabs, but if you're in the same document,
0:51:36 you probably want to see the same thing.
0:51:38 So it's the same that
applies to a music app.
0:51:41 Maybe in one tab you want to have.
0:51:43 The playback of one track and the another
one, you want to not have the same
0:51:48 playback, otherwise you hear it twice.
0:51:50 but you want to maybe work on a playlist.
0:51:53 And so keeping things in sync is
important, but I don't want to,
0:51:58 constantly as the playback progresses,
have persistent events for this.
0:52:02 So I try to A, have like, very
Deliberately small events.
0:52:08 And the other thing is where I have
events that are broadcasted around.
0:52:12 But, if the app reloads, it
doesn't rehydrate from those.
0:52:16 It either catches them midway
or it's not important enough.
0:52:21 that it shows it so very similar
to the presence feature in Figma.
0:52:25 So I have implemented a first version
of this, but I think there can be
0:52:29 use cases where you might want to
keep them around for like 10 minutes
0:52:34 or 10 seconds, like you say, and
then have a version of compaction.
0:52:37 I think that that's really interesting.
0:52:40 What you're describing sounds really cool.
0:52:41 I'll be interested to
see this code someday.