localfirst.fm

A podcast about local-first software development

Listen

Conference

#24 – Ben Holmes: Astro, Simple Sync Engine & Warp

The guest of this episode is Ben Holmes, a senior web developer and educator known for his whiteboard videos. After having spent most of his career building server-centric applications, Ben recently explored local-first software by building a simple sync engine which we talk through in-depth. 

Mentioned in podcast:

Links:

Thank you to Jazz for supporting the podcast.

Transcript

Intro
00:00What if I'm on my phone, I type a thing and then I go back to
00:02my computer and I try to see it?
00:04Is it gonna merge it together?
00:06Is it gonna break?
00:07I have no idea because.
00:10Client side stores don't help you with those problems.
00:12They're not designed for those problems.
00:14They're designed for like ephemeral state that can disappear at a moment's
00:18notice if it needs to for some reason.
00:21so that led me down exploring local-first technologies and I found pretty quickly
00:26how capable SQLite is for these use cases, including in the browser.
00:32Welcome to the localfirst.fm podcast.
00:34I'm your host, Johannes Schickling, and I'm a web developer, a
00:37startup founder, and I love the craft of software engineering.
00:41For the past few years, I've been on a journey to build a modern, high quality
00:45music app using web technologies, and in doing so, I've been falling down the
00:49rabbit hole of local-first software.
00:52This podcast is your invitation to join me on that journey.
00:55In this episode, I'm speaking to Ben Holmes.
00:58A senior web developer and educator known for his whiteboard videos
01:03after having spent most of his career building server centric applications.
01:07Ben recently explored local-first software by building a simple sync engine
01:11from scratch before getting started.
01:14Also, a big thank you to Jazz for supporting this podcast,
01:18and now my interview with Ben.
01:20Hey Ben, so nice to have you on the show.
01:22How are you doing?
01:24Hey, I'm doing great.
01:26Yeah, how are you doing?
01:27I'm doing fantastic.
01:29Super excited to have you on the show.
01:31I'm very certain that most of the audience already are familiar with who
01:34you are since you have quite a reach on, Twitter, x, other platforms, et
01:40cetera, where I think you're doing an excellent job of taking like novel
01:45concepts and breaking them down in a very simple and approachable way.
01:49And I think you've done the same for some local-first related topics recently.
01:54So maybe some of the audience are already familiar with your work in that regard.
01:59But, for those who are new to you, would you mind giving a background who you are?
02:04Yeah, totally.
02:05I'd be shocked if everyone knows, but if you've seen a guy with a whiteboard
02:09around the internet, it might be me, especially if it's a vertical video.
02:13so I've been doing a lot of work in just the engineering space for a long time.
02:20So been at Astro for a few years and we've been building the
02:26framework for content sites.
02:28And I know actually you have some experience working with Contentlayer
02:31and other things, but content sites are a nice place to get started
02:34because they're a nice, well-defined use case for web technologies and
02:39Astro was trying to spearhead being like the simplest way to do that.
02:43so was able to contribute to that for a long time and also produce a lot of
02:48videos on learnings in the process.
02:51So it started with taking a rock band microphone and a whiteboard outta my
02:55closet and just recording stuff and just seeing where it went and been doing it for
03:00years now and kind of honing the craft.
03:03And we've covered all sorts of topics, including, different
03:07web frameworks, Tailwind tips, database providers using SQLite.
03:12Most recently putting SQLite in the browser, which is gonna be
03:16very relevant today, I'm sure.
03:18and most recently I've made the jump to Warp Terminal, so I'm gonna
03:23be, well, I have joined their team as a product engineer and we're
03:28sort of spearheading the future of bringing AI and agent workflows to
03:33everything you do in the terminal.
03:35So if you forget a command, need to kill a port, need to go through
03:38a full get workflow or even edit files in your projects, you can just
03:43ask Warp to do that for you or fall back to all the features that make
03:46Warp just a really nice terminal.
03:48It's a really great fit.
03:49Been working with them for a while and now get to do that full time.
03:53We'll continue to do all the videos that you see around the internet.
03:56That is awesome.
03:57I'm certainly coming back to Warp since I wanna learn more about that as well,
04:01but taking things one step at a time.
04:04you've been digging into local-first related, you've been mentioning
04:08SQLite running SQLite in the browser.
04:10Looking into that over the course of the last year or so.
04:14I'm very curious, like what led you to explore that since with Astro, et
04:19cetera, you're probably, that's like a different part of building websites,
04:24really where you maybe like Astro is famous for introducing the island
04:28architecture where you go like very light on bringing JavaScript into the, website.
04:35And if it's almost like the other extreme where you want to bring a lot of
04:40JavaScript into the web app, deemphasize the server part, and also more on
04:45the spectrum from website to web app.
04:48Certainly much heavier on the web app spectrum.
04:51So yeah.
04:51Can you share more what led you to, to this path?
04:55well the past year has been kind of like an existential crisis of how are you
04:59supposed to build websites, and I think we all went through that as an industry,
05:03as things kind of shifted back from client side, heavy react apps towards things
05:09that are more server rendered and people are slowly trying to question that idea
05:14again and see what we can learn from storing more information in the client
05:19and sinking it back to your servers.
05:21So I also noticed a wave that's completely opposite.
05:26Of local-first apps, which would be something like HTMX, where everything is
05:32purely server driven, state stateless.
05:36Everything is from like the REST API protocol.
05:40And there's a really nice simplicity to it where a state lives in one
05:45place, the server's a source of truth for everything, and you use different
05:50return values for each HTML sheet to decide what is going to render next.
05:55And you accept that there's going to be network latency for most interactions
06:00on the site, except for very small dropdown toggles and the like.
06:04But anything that involves state is always going to go back to
06:07the server and map it back.
06:09That obviously has trade-offs that people have tried to get away from
06:12with client side architectures.
06:14But the reason it's so nice is you don't have to think about, you have
06:18this server state, you have this client state, and you're constantly trying
06:22to keep them sort of melded together.
06:24And I think that's something that a lot of server side rendered applications have
06:30run into most recently react trying to add on, like use optimistic hooks and
06:37libraries like TRPC, letting you show data optimistically and then replace it
06:41with the server value when it comes in.
06:43And these are important principles, but it's a lot of manual effort to first send
06:49a request to the server and also keep the client in some optimistic version
06:53of that to cut down on network latency.
06:56You're having to pull levers on both sides and you don't really
06:58know where the truth lives.
07:00So, from my experience that's led to a lot of manual work, writing and
07:06rewriting local stores, maybe with Redux back in like the mid 2010s now
07:11using, query hooks, even GraphQL if you're racing for those kinds of tools.
07:16So it's, very messy using those things that are isomorphic, AKA, things that run
07:23both on the server and on the client, and trying to think of it in the same way.
07:28So one response to that is just, we don't need client side JavaScript.
07:32We're gonna do everything on the server.
07:34And that's very easy to understand because it goes back to like how
07:38rest was designed in the eighties.
07:40And then there's the other reaction, which is the, to other side, the
07:43client is the source of truth for pretty much everything that's going on.
07:46And the server's just a broker to keep different clients in sync
07:50and to push changes so everyone can stay up to date or even doing
07:54decentralized servers if you go further.
07:57But it's that other side of the coin.
08:00Of we want one source of truth.
08:02We don't want to think about, we have this server and this client and we constantly
08:05have to write logic to glue them together.
08:07No, you either store all the state on the server, kind of like an HTMX style,
08:12or you store all the state on the client using local-first technologies.
08:16And having explored the first one for a long time, since Astro is meant to be
08:21like a static website or server driven website, I was excited to explore the
08:26other side of storing everything on the client, keeping it up to date, and then
08:30figuring out how synchronization happens.
08:33That's also been around for a long time.
08:35If you look back to just like the first calendar app on your phone or on your
08:40computer, this challenge has been around since probably SQLlite was created.
08:44but it's only now that web devs are starting to get a lot of footholds
08:48to also apply this to websites if that's something that you need.
08:53I think you've summarized it really well.
08:55And, I would go even as far as saying there is an elephant in the room that
09:00most people are aware that there is an elephant, but they really don't
09:04have yet the, the right terminology.
09:06And I would say the term here of the elephant is distributed systems.
09:10We have distributed states and distributed systems are really, like,
09:14that's a core discipline of computer science that doesn't really have
09:19just like the good answer, but it's a really, really hard problem alongside
09:24of like naming things and so on.
09:26But, that part, like everyone who's building a web app,
09:30anything that has state.
09:32On the server, like even in the server, typically you have by definition
09:37also already a distributed system where you have states in your API
09:41throughout the request lifecycle, but then you also have it in the database.
09:45You might have like concurrent requests.
09:48So there you already have a distributed system.
09:50Typically there it's much less bad because they're like, you can just
09:55trust the server already less.
09:57So you just trust the database.
09:59So you basically push it all down to the single choke point.
10:02But if you wanna trust the client even more now all the distributed
10:07parts are get the distance grows and therefore the divergence.
10:12And I have in university, when I studied computer science, I don't
10:15think I've actually taken a class on distributed systems and I've missed a
10:20memo where someone would've instilled it in me is like, Hey, everything
10:24that are you gonna build will suffer from distributed systems, make sure to
10:28understand this problem properly and then design the architecture around it.
10:31And I think most web devs are not aware of that.
10:35And what you've just laid out, I think is exactly suffering from this problem.
10:41And this is what I think where some framework creators are making
10:45the very smart decision to empower the server as much as possible.
10:50Because if you live by that sort of maxim, at that point, the implied
10:57damage that you can cause by building it in a certain way is minimized.
11:02And, my journey over the last five years or so has been almost like
11:06intentionally letting the pendulum swing to the most other extreme where all the
11:11state is governed by the client, since I'm pretty convinced that the middle
11:16ground is just pain and suffering.
11:19So I'm pretty convinced that you should really analyze the use case that you have.
11:24And if it's a daily driver productivity app, you probably wanna move as much
11:29state to the client as possible.
11:31Where it happens, like you produce most of your state in your calendar app or in
11:35your Notion, or in your other notes app or whatever you want to have it declined.
11:41And, if it's something like New York Times, then you don't
11:45produce any of that date.
11:46So it should be on the server.
11:48And, I, feel like more people should, start with that assumption, then
11:52build an architecture around it.
11:54But, yeah, I think you summarized it super well.
11:58Yeah.
11:58And it is tough.
12:00To prescribe either side because there are neat buckets like content sites
12:06and client side apps like Notion.
12:09But the classic example is, well what about e-commerce where things are
12:15server driven until you're in the add to cart flow and now it's client driven.
12:18So what do you do then?
12:19How do you architect it?
12:21We actually met this challenge building the Astro storefront front template,
12:26which was mostly server driven with some client side components sprinkled in.
12:32I think the answer there was, it's still fine to leave things server
12:36side and not rely on optimistic updates too much, except for like
12:42small examples like increase and decrease quantity in your cart.
12:47Do you want that to feel instant and happen on a delay or a debounce but
12:51everything else is server driven.
12:53There is no perfect architecture, I guess is what I'm saying.
12:56It's more painful when you really try to blend it 50 50 and nail every use case.
13:01But if you can keep it 80 20, where 80% is in one realm and 20% is business logic in
13:07the other realm, like 80% server driven, 20% client side complexity or less, then
13:13you're kind of minimizing the footprint of pain that you could run into since I've
13:18certainly noticed that with some of the newer patterns and older patterns with.
13:23Client side apps.
13:24but there was a second thing you mentioned about distributed
13:26systems that I, totally agree with.
13:29I was reading the book Designing Data Intensive Applications, the big
13:35O'Reilly book, probably the first one you find looking up like CS principles.
13:40And I think there was one section about distributed systems where it walked you
13:44from like single leader replication.
13:47You got one database and if you want to have some caches to make reads
13:52faster, you just put all your rights to one database and then it'll replicate
13:56the reads out to everyone else.
13:58So you can't write to a bunch of different regions.
14:00You can write data to one region and then it'll sort of.
14:04Push out all of the clones and allow eventual consistency to work.
14:09but there are cases where that doesn't work anymore.
14:12Like in Notion, I cannot wait for the server to update the document.
14:16It's just gonna update when I'm typing and when I add blocks and when people
14:19are invited to join the document.
14:21All that stuff matters.
14:22So you need some way to be able to write to maybe the nearest node.
14:28Like if you go fancy with CloudFlare, you could have really localized,
14:31durable objects that are two feet away from your computer and that reduces
14:35latency in some sort of magical way.
14:38but then it kind of explains well if you just put a data replica just on the
14:43client device, that's another version of multi leader replication, where
14:49instead of replicating in some really edge node, you're just replicating
14:53on the person's computer and everyone is like a mini server unto itself,
14:57where you just read and write data and they'll all report back to some
15:01centralized source of truth later on.
15:04So when you're working with, like, is it a distributed system?
15:07It is.
15:08Even when it's running on your device, if there's some sort of synchronization
15:12layer, you're just moving it closer and closer and closer to the computer
15:15until it iSQLiterally in the computer.
15:18It's inside the house.
15:20I've, heard a, very interesting framing of sort of like that
15:24elephant in the room and like that problem that we just talked about.
15:27I think it's by Carl who worked on SQLSync and recently, released Graft,
15:32which is another fascinating project.
15:35And he has written a blog post about, don't recall the exact name we're gonna
15:39put it in the, show notes, but it was basically along the lines of like, your
15:43application is a database and that's bad.
15:47I gotta look it up.
15:48what the, exactly the title was.
15:49But it was basically, he's making the point that every app that is sufficiently
15:56getting more complicated will basically build its own version of a ad hoc database
16:03and if you're just using React useState or something else enough, and then
16:08you try to add persistence, et cetera.
16:10What your app is basically becoming is a poor implementation of a database
16:15where like all the things that a database does a lot of like R&D great
16:21work to make it fast, to make it correct, to make it like nicely, like
16:27transactionally correct, et cetera.
16:29All of those things you're now trying to handle through, useState and useEffect
16:35and like when this thing changes, also change that thing, et cetera.
16:40And, I just thought that framing was so elegant and, his conclusion, which
16:45I agree with is like, hey, if let's try to make the app about what the app tries
16:50to do and let's leverage a database so we can focus on the actual like
16:55features, et cetera we want to implement.
16:58And I think that's another kinda articulation of the elephant in the room
17:03that we're accidentally and without being aware of it Building databases as that
17:09are sort of camouflaged as apps and, we should embrace it more if we're starting
17:14to see those signs of those problems.
17:17So you've mentioned that you've gone down this path, a lot out of curiosity and
17:22just because you've, pretty exhaustively explored the more server centric
17:27paths, can you share more about like what were your frustrations and pain
17:31points that you felt where you wanted to go more, like embrace the client
17:35more, but still trying to do that with the more server centric architecture.
17:41So you've mentioned optimistic state, et cetera, maybe can motivate
17:45this through a concrete app you wanted to build where you just felt
The pain points of embracing the client
17:51Yeah, I mean I've played with all sorts of side projects as we all have.
17:56and I was working on a few different things.
18:00One was like a localized note taking app, and I was also playing with local
18:06LLMs and other pieces to add vector search into a local environment because
18:12I was running up on the limits of using Notion and being very frustrated with
18:17loading spinners and offline warnings when I was trying to use the app.
18:21I hear that's being changed and rectified as they build out their
18:26own, like SQLite replication.
18:29I know that's in there.
18:29There's some fascinating videos about that on the internet.
18:33but I still do see the value of just open up Apple Notes, you type things and it's
18:39just kind of there in a SQLlite store.
18:40You can find it on your file system.
18:42All your Apple Notes are just in a SQLlite store and it's great.
18:45so I thought, well, it seems like that's the answer.
18:48If I were to go the traditional server route, I would sit here
18:51waiting for all these updates to persist, which just wouldn't work.
18:55Or I'd be sort of gluing together a bunch of useState calls and figuring
18:59out how do I update this server and what if someone else updates it?
19:02What if I'm on my phone, I type a thing and then I go back to
19:04my computer and I try to see it?
19:06Is it gonna merge it together?
19:08Is it gonna break?
19:09I have no idea because.
19:12Client side stores don't help you with those problems.
19:14They're not designed for those problems.
19:16They're designed for like ephemeral state that can disappear at a moment's
19:21notice if it needs to for some reason.
19:23so that led me down exploring local-first technologies and I found pretty quickly
19:28how capable SQLite is for these use cases, including in the browser.
19:34you can load up SQLite with like a wasm build as you're
19:38very aware and use it even with.
19:41Like very nice SQLite libraries.
19:45If you wanted to use Drizzle for example, which is a common SQL querying library
19:50and JavaScript, it matches onto the browser version of SQLite perfectly.
19:55So you can actually make declarative, find many notes and
19:59it'll just do the little SQL query.
20:01It'll join it up with all the authors of the post and it'll
20:04just give it back to you.
20:05Kind of like you're on the server, but you're on the client.
20:08And you can use client side ORMs or query builders, whatever
20:12your flavor of preference.
20:14So that was very empowering to see.
20:16Yeah, you can bring all of these niceties you get from server side data
20:20querying and bring it into the client.
20:23And I tried to stretch it a little bit further by asking,
20:26what about SQL extensions?
20:28Could I add a vector search plugin, for example?
20:32The answer is yes.
20:33There actually is a vector search plugin that I think is developed
20:36by someone on the Mozilla team.
20:38So it is pretty battle tested in different languages.
20:41I think they're sponsored, yeah I've had the chance to meet them in October
20:47when I was in LA super lovely person.
20:50And, they have some sponsorship from the Mozilla team currently.
20:55Nice.
20:56Yeah, and I was playing with the, rust flavor of that since, well now
21:01I'm working at Warp, and Warp is a rust powered terminal, so naturally I
21:05need to get up on the Rust knowledge.
21:08And also if you build apps with Tori, which is a native desktop application
21:13tool that uses Rust as well, so.
21:16Side tangent, but it is nice to use tools that could work in JavaScript as well
21:19as rust in very efficient languages.
21:22so I reached for that.
21:23I put vector search in.
21:24I was also able to run an entire LLM across the data in the
21:29browser and just load up like the entire vector search setup thing.
21:35and I just said, all right, I'm gonna backport all of my
21:38markdown files into this thing.
21:40I'm gonna vectorize all of them in the browser.
21:42I'm gonna search them in the browser.
21:44And I was able to get it working with like next key search.
21:48And it was absolutely mind blowing.
21:49Like this is like an actual AI powered search tool.
21:53And you can get like with every keystroke new results
21:57And is isn't that wild like that this machine that we have like sitting
22:01on our laps or like this machine here in my hands that is capable
22:07of all of those things and just the way how we kind of build web apps.
22:11Over the last decade or so, we've kind of forgotten or denied the
22:16capabilities of our client devices.
22:18And we only just like trust the server and we've almost like, why did no
22:22one tell us that this is possible?
22:25And that's so magical when you see that this is working and just the
22:28stuff that's currently, in the works with like web, LLM, et cetera, where
22:32it can run like a full blown like Llama model locally in your browser.
22:37running on web GPU is absolutely wild to the capabilities that we have.
22:43But I think what's holding us back is where does our data
22:47live and everything else.
22:48Kinda like, it's almost like a second order effect from where the data lives.
22:52And this is what, your anecdote really nicely highlights of like, you go with
22:57SQLite with your data and then you like bring in another superpower of like SQLite
23:02Vec with the vector embedding, et cetera.
23:05And I think it all starts with where the data is, how your
23:08application is being shaped.
23:10Yeah.
23:11And it is good to find that.
23:13Just common data layer.
23:15SQLite is the easy answer.
23:17PG light is a more robust exploration of bringing Postgres to local devices.
23:22That's a bit earlier on.
23:24but I think we're entering a world where software is just so easy to spin
23:28up that you will very quickly have a web client, a mobile app client,
23:33a desktop client, and they're all talking to the same sync server.
23:38And when you're in that world, it's nice to just reach for SQLite.
23:40'cause I can run SQLlite on my iPhone.
23:42I can run SQLlite on my Android device, I can run it in the browser and I
23:46could run it in a desktop application.
23:48So as long as you just have this concept of clients write to SQL servers and
23:54those SQL servers have some way to talk to each other, then you can build
23:58these multi-platform applications very quickly, even across different languages.
24:04And just figure out what that sync layer looks like, which
24:07we can probably talk about.
24:09that would've been my next question since, I think what you started with
24:12was probably without the sync part yet, where you can just locally in
24:17your browser web app, you successfully ran SQLite using the wasm build.
24:23Then you brought in SQLite Vec, you could, get the AI magic to work.
24:27You saw like how insanely fast everything feels.
24:30You write something, you reload the browser.
24:33Even if you're offline, it's all still there.
24:35Great.
24:36But now you're thinking, okay, it works on like local host 3000.
24:41how do I get it deployed and how do I get it so that, if I accidentally
24:47open this in a cognitive tab and I close it, poof, everything is gone.
24:52and how do I get it show up on my, phone so this is kind of
24:56collaborating with yourself, but on also collaborating with others, which
25:00we maybe punt on that for a moment.
25:03But yeah.
25:03How did you go from, it works locally almost like if you use like just
25:09local storage to trying to share the goodness across your devices.
From working locally to working across multiple devices
25:15Well, I think we both have the privilege of just ask Andrew on the Rocicorp team
25:19and he'll point you to some resources.
25:21if you don't have that, I do recommend that.
25:25Well, the one approach that I used was just reading through the Replicache docs.
25:31That explain how their sync engine works on a very high level.
25:35And Replicache is a batteries included library, well some batteries included
25:42library that sets up a simple key value store for you where you can write and read
25:48keys on the client and you can author code that will post those changes to a server
25:54that you own and pull changes back from the server whenever changes are detected.
25:59And you can implement that on a SQLite or anything else if you just have
26:04their simplified mental model, which is very inspired by just how Git works.
26:09So in Git, if you had to change locally and you wanted to push it up to the
26:14main line, you would run Git push.
26:17And the same kind of thing happens with their sync engine service, where
26:21anytime you make a change locally and you can decide what that means,
26:25maybe it's a debounce as you're editing a document, maybe it's when
26:30you click on a status toggle in Linear and you wanna change that status.
26:34Whenever the trigger is, you can use that to call push, which would probably call
26:40an endpoint written on your own server.
26:43That's simply a push endpoint and that can receive whatever event you ran
26:49on the client so that the server can replay it, back it up into its own
26:53logs and tell other clients about it whenever they try to pull those changes
26:58back down so you can run your own little push whenever you make a change.
27:03And then other clients on an interval or a web socket or some
27:07other connection can pull for data whenever they want to get the most
27:11recent changes made by other people.
27:14Now the question would be, what am I pushing and what am I pulling?
27:18Like what data?
27:20Needs to be sent across.
27:22And there are a couple different methods.
27:25I know Replicache uses a data snapshot mechanism.
27:30I don't fully know the intricacies of it, but I know that because they use
27:35like a key value storage, which is much simpler than like a full-blown database.
27:39They can take a snapshot of what the server looks like right now and then the
27:44client has its own copy of that server snapshot and it can just run a little
27:47diff the same way you'd run a get diff to see what code changes you have made.
27:52It can run that diff and you can see, all right, I added this key.
27:55All right, updated this status flag, and then tell the server,
27:58this is the diff, this is the change that was made to the data.
28:02And the server can receive that request and say, okay, this is the change
28:06that I need to make against my copy.
28:08And then I will tell other clients to also apply that diff whenever they pull.
28:13it's a very Git inspired model, and it works if you're able to diff data easily.
28:19If you are working with like a full blown SQLlite table, you run table
28:23migrations, tables, change shapes, that's a very hard thing to keep track of.
28:27So another option that I implemented for the, learning resource I
28:33created called Simple Sync Engine.
28:35If you find that on my GitHub, probably in the show notes, you can see that.
28:40But it was meant to be a very basic implementation of this pattern
28:43that uses event sourcing instead.
28:46So rather than sending a diff of how the data should change, instead sends
28:52an event that describes the change that is made like a function call or
28:57an RPC, however you wanna think about it, where you would tell the server
29:01I updated the status to this value.
29:05And in our implementation we create these little like function helpers that
29:09can describe what those events are.
29:11So you might have like an add status event.
29:13So like the type of the event is add status and that accepts a few arguments.
29:19It accepts the status you wanna change it to, and the ID of the
29:22record that has that status currently.
29:25So the server receives that event, it sees, okay, the type was set status.
29:30I see two arguments here.
29:31The ID of the record.
29:33And the new status that should be applied.
29:35I'm gonna go ahead and run that event as a database update.
29:39So it has that mapping understanding of, I received this event, I know that
29:43maps to this SQL query, so I'm gonna go ahead and make that change on my copy.
29:48And then whenever people poll, you can send that event log out, or you
29:52can send whatever events the client hasn't received up until that point.
29:57You can kind of think of those like Git commits where you're pulling
30:00the latest commits on the branch and the server's able to tell you, here
30:04are the latest events that were run.
30:06Go ahead and run those on your replicas so both the server
30:10and client know what events.
30:12Actually mean, like this event means I need to make this SQL query.
30:16And it's able to do that mapping.
30:17and you can have a bit of freedom there.
30:19If you had like a very specific kind of data store on the server, like
30:24MongoDB, you could customize it to say, whenever I receive this event, it
30:28means this MongoDB query and this call to the Century error logging system,
30:33or whatever middleware you wanna do.
30:35As long as server and client agree on what events exist and what changes they
30:40make in the data stores, then everyone can be on the same page whenever
30:44they're syncing things up and down.
30:46you're very familiar with event sourcing as well.
30:49I'm curious if there's things that I've missed or important edge cases
30:53that we should probably talk about.
30:55I think you very elegantly, described how simple the foundation of this can be and
31:00hence the name of like Simple Sync Engine.
31:03I think this has served as a great learning resource for you and
31:06I'm sure for many, many others.
31:08And once you like, start pulling more on that thread, you realize, oh shit like,
31:13okay, that thing I didn't think about.
31:15Oh, what about this?
31:16So you've mentioned already the reason why you preferred, event sourcing over
31:22the snapshot approach because like with SQLlite, What would you actually compare?
31:27This is where I would give a shout out again, to Carl's work with Graft.
31:32this is what he's been working on.
31:33We should have him on the show highlighting this as well.
31:36But, where he's built a new kinda sync engine that, is all about that
31:41diffing of like blocks of storage.
31:45I think all focused around SQLite and that gives, that does the heavy work
31:50of like, diffing and then sending out the minimal amount of changes
31:55to make all of that efficient.
31:56Since there's this, famous saying of like make it work, make it right
32:02or correct and then make it fast.
32:04And working on all of this has really given me a deep appreciation for all
32:09of this since like, and I'm sure you probably also went through those stages
32:15with the Simple Sync Engine, like with making something work in the first place
32:19was already quite the accomplishment.
32:21But then you also realize, ah, okay, in those cases this is not quite yet correct.
32:25And then you could go back and like try to iterate and then you also realize,
32:30okay, so now it has worked for my little to-do app, but if I, now, depending
32:36on the architecture, if I roll this out and put in hack news and suddenly
32:40have like 5,000 people there on the same time, this thing will break apart.
32:45Will A not be correct in some ways you didn't anticipate and
32:48will also not be fast enough.
32:50So now making this fast.
32:52It is at the end of the day is like really reinventing.
32:55It's like your app is becoming a little database and you want
33:00to like move as much of that database burden to the sync engine.
33:05This is why folks like Rocicorp, ElectricSQL, et cetera, they're
33:09doing a fantastic job trying to absorb as much of that complexity.
33:13But building something like this by yourself really gives you an understanding
33:18and an appreciation for what is going on.
33:21I love the Git analogy that you've used, but just a, a couple of
33:25points just similarly to how, your sync engine works is actually very
33:30analogous to how the Livestore architecture on a high level works.
33:36But I've had to, before I arrived at that, I really wanted to think through a lot
33:42of like the more further down the road.
33:45Like, what if situations since, one that I'm curious whether you've
33:50already run into, whether you resolved in some way or left for the
33:53future, is how would you impose a total order of your change events?
33:59So this is where When you have, like, let's say this to-do app
34:04or like a mini Linear, app.
34:06Let's say you create an issue and then you say you, complete the issue or
34:12you toggle the issue state, et cetera.
34:14It can mean something very different, if one happens first and then the
34:19other, or the other way around.
34:21And for that, where you have your events of like, hey, the issue was created,
34:26the issue status was changed, this, the issue status was changed, that,
34:30the order really matters in which way in which order everything happened.
34:36And, this might be not so bad if you're the absolutely only person using your
34:40app and typically only on one device.
34:43But when you then do stuff between multiple devices or multiple
34:46users, then it's no longer in your single user's control.
34:50That stuff happens concurrently.
34:53And then it really matters that everyone has the same order of the events.
34:58And this is where you need to impose what's called a total order.
35:02And I'm very curious whether you've already, hit that point, where you thought
35:06that through and whether you found a mechanism to impose it since there's many,
35:10many downstream consequences of that.
Imposing a total order
35:13Right.
35:13And I definitely did hit it and.
35:16You will find in the resource, it's not addressing that issue right now.
35:20It's in a very naive state of when change is made, send, fetch, call to server.
35:26And there are a few problems with that, even if you're the only client.
35:30Because first off, if you send one event per request, it's very possible that you
35:35send a lot of very quick secession events in like an order, and then they reach the
35:41server in a different order because maybe you pushed the first change on low network
35:47latency, the next one on high latency, or actually the reverse of that where
35:50the first change hits after the second change because that's just the speed of
35:54the network and that's what happened.
35:56And also you need to think about offline capabilities.
36:00If you push changes when they happen, how do you queue them
36:04when you're not connected to the internet and then run through that
36:07queue once you're back online?
36:09That's another consideration you kind of have to think about.
36:12Could be solved with just like an in-memory event log and
36:15just kind of work with that.
36:16But you still have the order issue.
36:19I'm familiar with atomic clocks as a method to do this.
36:23There are even SQLite extensions that'll sort of enforce that, having
36:28not implemented atomic clocks.
36:30Is it kind of this silver bullet to that problem or are there more
36:34considerations to think about than just reaching for something like that?
36:38Right.
36:39I suppose you're referring to vector clocks or logical
36:42clocks on a more higher level?
36:43Yeah.
36:44since the atomic clocks, at least my understanding is like that's
36:47actually what's, at least in some super high-end hardware is like
36:51an atomic clock that is, like that actually gives us like the wall clock.
36:56So Right, right now is like.
36:58Uh, 6:30 PM on my time, but this clock might drift, and this
37:03is what makes it so difficult.
37:04So what you were referring to with logical clocks, this is where it basically,
37:09instead of saying like, Hey, it's 6:30 with this time zone, which makes
37:14everything even more complicated, I'm keeping track of my time is like 1, 2, 3.
37:20It like might just be a logical counter, like much simpler
37:24actually than wall clock time.
37:26but this is easier to reason about and there might be no weird issues of
37:31like, Daylight saving where certainly like the, the clock is going backwards
37:36or someone tinkers with the time, this is why you need logical clocks.
37:40And, there, at least the mechanism that I've also landed on to
37:44implement, to impose a total order.
37:47But then it's also tricky, how do you exchange that?
37:50how does your client know what like three means in my client, et cetera?
37:54And the answer that I found to this is to like that we all trust.
38:00A single, authority in the system.
38:02So this is where, and I think this is also what you're going for, and with the Git
38:07analogy, what we are trusting as authority in that system is GitHub or GitLab.
38:13And this is where we are basically, we could theoretically, you could
38:17send me your IP address and I could try to like pull directly from you.
38:20It would work, and that would also work with the system that you've built.
38:25However, there might still be, they're called network petitions,
38:29where like the two of us have like, synced up, but some others haven't.
38:33So as long as we're all connected to the same, like main upstream node, that
38:39is the easiest way to, to model this.
38:41An alternative would be to go full on peer to peer, which makes everything
38:46a lot, lot, lot more complicated.
38:49And this is where like something, like an extension of logical clocks called
38:53vector clocks, can come in handy.
38:55you've mentioned the, the book, designing dataset intensive application by Martin
39:00Kleppman had him on the show before.
39:02he's actually working on the version two of that book right now, but he's also done
39:06a fantastic free course about distributed systems where he is walking through all of
39:12that, with a whiteboard, I actually think so, I think does what, what the two of
39:18you have very much like you've both nailed the, craft of like showing with simple
39:24strokes, some very complicated matters.
39:27so highly recommend to anyone who wants to learn more there.
39:31Like, learn it from, from Martin.
39:33He's, like an absolute master of explaining those difficult
39:37concepts in a simple way.
39:40But, yeah, a lot of things go kind of downstream from that total order.
39:45So just to, go together on like one little journey to understand like a downstream
39:51problem of this, let's say we have implemented the queuing of those events.
39:56So let's say you're currently on a plane ride and, you're like.
40:00Writing your blog post, you're very happy with it.
40:03You have now like a thousand of events of like change
40:07events that captures your work.
40:09Your SQLite database is up to date.
40:12but you didn't just create this new blog post, but you maybe while you're still at
40:16the airport, like you created the initial version with it with like TBD in the body.
40:21And your coworker thought like, oh, actually I have a lot of thoughts on this.
40:26And they also started writing down some notes in there.
40:29And now, the worlds have like, kind of drifted apart.
40:33Your coworker.
40:35Has written down some important things they don't want to lose,
40:38and you've written down some things you are not aware of the other ones
40:42neither are they, and at some point the semantic merge needs to happen.
40:48But how do you even make that happen in this sync engine thing here?
40:52And this is where you need the total order, where you basically, in the worst
40:57case, this is what decides, like who, gets a say in this, who gets the last say, in
41:04which order those events have happened.
41:07The model that I've landed on, and I think that's similar to what Git
41:12does with rebasing, is basically that before you get to push your own stuff,
41:18you need to pull down the events first, and then you need to reconcile
41:22your kind of stash local changes.
41:26On top of the work that whoever has gotten the, who got lucky enough to push
41:32first without being told to pull first.
41:35So in that case, it might have been your coworker because they've
41:39stayed online and kept pushing.
41:41And now it sort of like falls on you to reconcile that.
41:46And I've implemented a, like an actual rebase mechanism for this,
41:51where you now have this set of new events that your coworker has
41:56produced and you still have your set of events that, reflect your changes.
42:01And now you need to reconcile this.
42:03So that is purely on the.
42:05Event log level, but given that we both, want to use SQLite now, we don't
42:12need to just think about going forward with SQLite, but we also now need to
42:17think about like, Hey, how do we go?
42:19Like in Git you have like, you have this stack of events, right?
42:24So you have like a commit, which has a parent of another commit, which
42:27has a parent of another commit.
42:29It's very similar to how your events and this event log look like, except it's now
42:36no longer just one event log, but you also get this little branch from your coworker.
42:41So now you need to go to the last common ancestor.
42:44And from there you need to figure out like.
42:46How do I linearize this?
42:49I've opted for a model where everything that was pushed once cannot be
42:53overwritten, so there's no force push.
42:55So you basically just get to append stuff at the end.
42:59But, in order to get there, you need to first roll back your own stuff, then
43:05play forward what you've gotten first.
43:08and then on top add those.
43:10And the rolling back with SQLite is a, thing that I've like put a lot of
43:15time into where I've been using another SQLite extension, called the SQLite
43:21Sessions extension, which allows you, per SQLite write, to basically, record
43:27what has the thing actually done.
43:30So instead of storing, insert.
43:33Into issues, blah, blah, blah.
43:35when running that, you get a blob of let's say 30 bytes, and that has
43:40recorded on SQLite level, what has happened to the SQLite database.
43:46And I store that alongside of each change event, that sits in the event log.
43:53And the very cool thing about this is, I can use that to replay it
43:57on top of another database, but to kind of catch it up more quickly.
44:01But I can also invert it.
44:03So now I have basically this like, let's say 20 events.
44:07And for each, I've recorded what has happened on SQLite level,
44:11and now I can basically say.
44:13When I need to roll back, I can revisit each of those, invert each of those
44:17change sets, apply them again on the SQLite database, and then I'll end up
44:22where I was before and that's how I've implemented rollback on top of SQL Lite.
44:27So this is as mentioned when you're going, down the, rabbit hole
44:32of like imposing a total order.
44:34There's a lot of downstream things you need to do that makes
44:37this even more complicated.
44:39But, from what I can see, you're, on the right track if
44:43you wanna pursue this further.
44:45Yeah.
44:45And I do have a rebasing mechanism in place in mind that's more,
44:52just kind of a sledgehammer.
44:53I got two SQLite databases in mind.
44:56in the same way that on Git you have like your local copy of the main line and your
45:00local copy of your work, there's always this local copy of Main, that's just
45:05whatever events have come from the server.
45:07So this is the source of truth that the server has told me about and that was
45:12something I forgot to mention earlier.
45:13Explaining all of this is the server is the source of truth.
45:16It has that main line of the order of all of the events, and that is
45:21what all the clients use to trust.
45:23But yeah, it has like that local copy, and then when it pulls from
45:27the server, it'll update that copy.
45:29It'll look at all the events that are kind of ahead in the client,
45:33and then it'll say, okay, I'm gonna roll back my client copy of my
45:39branch to whatever the server is.
45:41And it's literally just a file right call.
45:43So it just overwrites.
45:46Your like client SQLlite file with a copy of the server one.
45:50And then we look at the events that the server didn't acknowledge yet
45:53and then we replay those on top as a very basic way to pull and make
45:58sure, because it's very possible that you made some changes locally that
46:02the server hasn't acknowledged yet.
46:04Like you've pushed them up still in process and you pull down the
46:08latest changes and you don't see all of that stuff that you pushed
46:11up yet because of network latency.
46:14So this sort of avoids that problem where you pull down from the server
46:18and now you need to replay whatever you did on the client that the
46:21server hasn't acknowledged yet.
46:23It hasn't received that network request.
46:25So that was a very basic need to have some rebasing, but it does
46:30get a lot more complicated when you have collaborators on a document.
46:34I've seen a few different versions of this.
46:37CRDTs is the fun, like magic wand.
46:40It does everything.
46:42but there are also solutions from Figma, for example, where they
46:47say everything in Figma is kind of its own little data structure.
46:50Like you can put some text and that's its own little data field.
46:54You have rectangles.
46:54Those are a data field.
46:56And whenever you update a rectangle, like you update the pixel width of
47:01a rectangle, that's like an update event on some SQL table that stores
47:05all the rectangles for this document.
47:07So whenever you make that update, it'll update the pixel value of whatever
47:12that row entry is, and then it'll push it up for other people to receive.
47:17And when you pull it down, it's last right wins.
47:20In other words, whoever the last person is in that order that the
47:24server decided on that total order.
47:26That's a new word I know about now.
47:28Didn't know it was called total order, but yeah, that, once you pull it down,
47:31whatever the server said was the order of events, that's gonna be the final
47:35state of that rectangle on your device.
47:38The only time it becomes a problem, and you may have experienced this, if you're
47:41ever working on like a fig jam together with a bunch of people, if you're all
47:45typing in the same text box, everyone's just like overriding each other and a
47:48text box glitches out and changes to whatever's on the other person's screen.
47:52You can't see people's cursors because you're fighting to update
47:55the exact same entry in the database and it can't reconcile those changes.
48:00so it only works up to, like you're editing different things
48:04in the file and you're not really stepping on each other too much.
48:08As soon as you're stepping on each other trying to edit like the same text field,
48:12then you wanna reach for something that's very, very fancy, like CRDTs.
48:17Which will try to merge elegantly all of the changes that you're
48:20typing into the same database field.
48:23It's maybe over-prescribed because of how powerful it is, but for those specific
48:28scenarios, it's really nice to reach for, and we can talk about them if you want.
48:32I only have a high level understanding of what CRDTs do, but it would be
48:36something to apply that kind of problem.
48:39my takeaway from where to apply, CRDTs versus where I would apply event sourcing
48:45is, CR DTs great for in two scenarios.
48:51One, if you don't quite know yet where you want to go.
48:54And where in the past you might've reached for, let's say, Firebase to
48:59just like have a backend of service.
49:00You know, you might want to change it later, but you just, for now,
49:04you just want to get going and, you can, particularly if you
49:08don't have like a strict schema across your entire application.
49:12So you just try to like, not go off the rails too much, but at least the
49:17data is like, mostly, like across the applications in a good spot.
49:22But as you roll this out in production, and, we are shipping
49:26an iOS app as well, that someone is, running an old version on.
49:31Now you don't quite know, oh, this document, this data document that has
49:35been synced around here, this might not yet have this field that the
49:39newer application version depends on.
49:42So now you have, like, this is where time drifts in a more significant
49:47way and in the more traditional application architecture approach
49:52you would, this way you don't trust the client in the first place.
49:54Then you have like your API endpoint and the APIs, versioned, et cetera, and
49:58everything is governed through the, API.
50:01But now you also need to tame that problem somehow.
50:03So at this point you're already, going a little bit beyond where I
50:07think CRDTs shine right now, which brings me to my next kind of more
50:12evergreen scenario for CRDTs, which are like very specific, tasks.
50:19And so text editing, particularly rich text editing.
50:22Is such a scenario where I think CRDTs are just like a very, very good, approach.
50:28There's also like, you can also use ot, like operational transform, which
50:32is, somewhat related under the covers, works a bit differently, but the way how
50:37you would use it is pretty similarly.
50:40And, related to rich text editing is also when you have like complex
50:45list structures where you wanna move things within the list.
50:49So if you want to go for the, Figma scenario, let's say you change the
50:55order of like multiple rectangles, like where do they sit in that layer order?
51:01how do you convey how you wanna change that?
51:04You could always, have like maybe an array of all the IDs that give
51:08you this perfect order, but if this kind of happens concurrently,
51:13then you need to reconcile that.
51:14So that's not great.
51:16And this is where CRDTs are also like a very, special purpose
51:20tool, which works super well.
51:23And so what I've landed on is use event sourcing for everything except
51:28where I need those special purpose tools, and this is where them reach
51:33for CRDTs or for something else.
51:35That's kind of the conclusion I, took away if you like the event sourcing approach.
51:41But, I think ultimately it really comes down to what is the application
51:46that you're building and what are, like, what is the domain of what
51:51you're building and which sort of trade-offs does this require?
51:54So I think in Figma.
51:56The real timeness is really important and it is recognized that those different
52:02pieces that are floating around, they're like pretty, independent from each other.
52:07So, and if they're independent, then you don't need that total order
52:10between that, which makes everything a lot easier in terms of scalability,
52:14in terms of correctness, and then you don't need to rebase as much.
52:18distributed systems is the ultimate case of it depends.
52:22and I think trying to build one like you did, I think is a very good way
52:28to like build a better understanding.
52:30And also I think that opens your eyes of like, ah, now I understand why Figma
52:35has this shortcoming or Notion if we are trying to change the same line, change the
52:40same block as where last writers, applies.
52:43Whereas in Google Docs, for example, we could easily change the, same word even.
52:49And it would reconcile that in a, in a better way.
52:52But, maybe you have some advice for people like yourself when you're
52:57just getting started on that journey.
53:00What would you tell people what they should do maybe shouldn't yet do?
53:05today 2025?
53:07There's more technologies out there now.
Beginner advice
53:12Depends on the type of learner you are.
53:13Sometimes some are very.
53:16outcome driven, like I need to see an app running in production for me to
53:21really get excited about this Tech.
53:24Other people are very first principles driven.
53:26Like I want to like screw in every nut and bolt myself to
53:30get excited about this thing.
53:32I tend to fall into the first camp where I think it is very useful to just
53:35look at the docs for something like Replicache and see how would you implement
53:39this kind of protocol step by step.
53:42Like how would you set up the event sourcing?
53:45How would you put the SQLlite store in the browser in the first place?
53:48Like what capabilities are there?
53:50And then try to think through those edge cases.
53:53As you run into them trying to build something, I use Linear as my sort
53:58of learning example, but you could use pretty much anything you want.
54:02so that's definitely one approach.
54:03Now there's just so many resources for how these things work under
54:07the hood that you can easily learn about the intricacies yourself.
54:11Another, resource is the talks given by who's the engineer at Linear.
54:18I think you've had him on the show.
54:19Tuomas?
54:20Yes.
54:21Yeah.
54:21He gave a few really helpful talks about how the Linear sync engine works on a high
54:28level, and that one's more opinionated.
54:30It reaches for technologies like MobX, which is a react specific state
54:36store, and also MongoDB for documents.
54:39but you still get a high level of how they think about the
54:41problem, which is really nice.
54:43the other option, if you're really results driven, you wanna
54:46see a local-first step running.
54:48You can reach for all sorts of frameworks and libraries at this point.
54:52Zero is the one that I've played with most recently, and it is Alpha Software you'll
54:58run into, it holds your hands, plugging in every battery and setting up everything.
55:03But error codes could be very confusing.
55:06but luckily their Discord is very welcoming and will answer any question
55:10that you have since their only goal is for everyone to get excited about
55:13the tech and use it in production.
55:15So I think Zero is a really great starting point as just, I wanna build an app.
55:20I'm gonna reach for a library.
55:22It will give you a query builder.
55:24So instead of writing raw SQL, it'll help you write SQL queries
55:28with some JavaScript functions.
55:30And it also works you through very common problems that you do hit at some point.
55:35And the big one is data migrations and, well, not data migrations, schema
55:38migrations, because when you have a data store on the client and you have a source
55:44of truth on the server everyone has to agree on how that data is shaped if
55:49you're using a SQL model and not something that's Firebasey as you were mentioning.
55:54So in those cases, you have to know like the four or five step
55:57process of update the server schema to add the new field, then update
56:02the client to add that new field.
56:04And then if you're trying to delete an old field for some reason, you would
56:08need to execute those on client, then server in the correct order, and then
56:13manage a database version so that if a client tries to connect with really,
56:17really old application code, the server can say, sorry, I only accept people who
56:22are on version five of this SQL schema.
56:26You're on version three, so I'm just gonna hard refresh your webpage and
56:29get you up to the latest version.
56:31all of these challenges are really interesting to think about and Zero
56:35helps you think through them out of the box and presents docs on all of these
56:40problems before you run into them.
56:42but I happen to be the type that wants to run into as many brick
56:45walls as possible without someone telling me what to worry about.
56:49I just wanna worry about it.
56:51so I think the Simple Sync Engine resource is great just because it
56:57doesn't do very much and there's a lot left up to the reader to go off
57:01and try to run into those challenges.
57:03I'm sure splunking through like the LiveStore implementation, I would
57:07find 50 ways that I could improve what I'm doing to get to that next step of
57:12like resilience, schema, migrations.
57:14I literally didn't even touch schema migrations.
57:17there's so much that you need to think about that just crawling
57:20through open source libraries is really, really helpful with, so
57:23that's my preferred learning approach.
57:25I just like going that way.
57:26I completely agree.
57:27And I also like, it's, it's sort of a bit of convincing yourself,
57:31is this entire thing worth it?
57:33And what I always appreciate if someone knows a little thing about
57:38me and then tells me, you know what?
57:41I don't think this is for you.
57:43I wouldn't hold anything back for someone who wants to look into this.
57:47to say like, this might not be what you're looking for.
57:49If someone is very happy with like building web apps with Vite Astro NextJS,
57:57et cetera, and they're productive, they're building this, e-commerce
58:02platform, or they're building a more static website, I don't think there's
58:08anything really where local-first would change their work situation.
58:14But if they're frustrated with like actual apps that they use day to day, when you're
58:21frustrated like yourself, when you're frustrated with Notion being too slow,
58:25et cetera, and you're building those more productivity daily driver apps yourself.
58:30For me that was like a music app.
58:32I got frustrated with Spotify and other music apps.
58:35I think this is the, right scenario where like local-first has something to offer,
58:40but, and I think it has also the potential to become a lot simpler and easier over
58:47has already become a lot simpler and easier or the past couple of years, and
58:51it's gonna be even more so in the future.
58:53And there will be use cases where it's actually simpler.
58:57To use local-first to build something, then using Next for something.
59:02but that won't apply to all scenarios.
59:05And so it is not a silver bullet.
59:07the closest thing you'll get to a silver bullet is the right architecture for
59:12the right application scenario, but by default there is no silver bullet.
59:17Neither is local-first.
59:19And I think someone should evaluate, Hey, is this even for me?
59:23that's, I think should be the starting point.
59:25Yeah, and a meta comment just because now I'm in the agent coding space, Warp
59:34is getting more capable by the day of actually editing files and scaffolding new
59:38applications for you, from the terminal.
59:41I've found it's less valuable to know the syntax of how all of these libraries work
59:47and a lot more valuable to just know high level, what are they doing, what's the
59:51architecture and how would I debug it?
59:53because these agents are very good at spitting out the syntax,
59:56if you draw a very clear picture.
59:59So if you go off and read designing data intensive applications, and you
1:00:03start diagramming to yourself how all of these systems are distributed, you
1:00:06could bring that diagram to Warp or just the cloud website if you want, and
1:00:12say, I wanna build this kind of app.
1:00:13Here's how the architecture's gonna work.
1:00:15This is gonna talk to this, and I know about this library.
1:00:19I know LiveStore uses event sourcing, so I would like you to implement that
1:00:24and use React, but follow the handrails because I understand the architecture.
1:00:29It'll give you a way better application than if you were to just say, give
1:00:32me a local-first app with React.
1:00:35It would probably maybe not struggle in the beginning, but definitely
1:00:38struggle as you try to figure out what it is doing or debug whatever sort
1:00:43of system level issues you're having.
1:00:45I fully agree.
1:00:46And given that the both of us are not just application developers but also
1:00:50tool creators, we spend a lot of time thinking about like, how do I leverage
1:00:55the degree of freedom that I have here in the API, the way how I design the API
1:01:00that is intuitive for someone that they like, ideally that this becomes like a
1:01:05pit of success where they intuitively use it the right way, but also if they
1:01:09use it the wrong way, how do they notice?
1:01:11Do they notice like as early on as possible through type safety or only
1:01:17if they're already in production and they felt like, wait, no,
1:01:20this, like, this was a path that I've wrongly taken six months ago.
1:01:24so you want to design all of this in a way that you like learn as early as possible
1:01:30whether you're in the right track or not.
1:01:31And I think you can't get better than simplicity than going for simplicity.
1:01:36And this is why I love the path that you've taken with the Simple Sync Engine.
1:01:42Through the push pull model because that's already, that is deeply familiar for
1:01:46developers and that is how we're using Git and that has really been proven.
1:01:51And there you can't really get much simpler than that.
1:01:54And I think simple is great for everyone.
1:01:57and once we have a simple foundation, we have a reliable foundation.
1:02:00We can build fast and nice things on top of it.
1:02:03But particularly mentioning AI systems, I make a lot of design trade offs
1:02:08now differently, where I care less about how much effort it will be
1:02:12to write or to discover that thing.
1:02:15Since we have now LLMs, do TXT, et cetera, I care a lot more about like,
1:02:20how does, how will you even spot where like this doesn't seem right.
1:02:24The robot has given me something weird and just doesn't match my,
1:02:29like, primitive understanding of how this entire thing fits together.
1:02:33And that should also help the robot to like not go in the wrong
1:02:37direction in, in the first place.
1:02:39So, yeah, I love that.
1:02:41Like, and going for a simple design decision, the simple like overall
1:02:47system architecture that's gonna help you as a future programmer, observing
1:02:52little robots, building things, a lot more to know what's going on.
1:02:57So maybe we use that as a last segue to hear a little bit more
1:03:01about what you're doing now at Warp in regards to agents, et cetera.
1:03:06I've been using Warp for, a little bit.
1:03:09I still use it, for my standalone, terminal, but most of my terminal
1:03:15work is also happening within Cursor, which is integrated in like the
1:03:20agent thing and Cursor, et cetera.
1:03:22Maybe can, yeah, help me a little bit of like how I bring those two together
Warp
1:03:29Yeah, and it's an interesting world of sort of agentic coding solutions.
1:03:35It feels like there's a new approach to it every other day.
1:03:38before joining Warp, I was also using Kline a lot, which is a VS Code
1:03:42extension that's fully open source that will, from my experience, give
1:03:47you a more quality agent output.
1:03:49Since it has these two phases, you can flip on a plan switch and
1:03:54it'll use a reasoning model to take whatever you tell it and turn
1:03:59it into like a step-by-step plan.
1:04:02And you could walk through like, no, that architecture doesn't make sense,
1:04:04or Let me upload this six cal draw.
1:04:06I actually want to work like this.
1:04:07And you can go back and forth on like a design doc and then you can
1:04:11flip it to act mode and that engages Claude to go build that for you.
1:04:15And it's very hit or miss actually doing like the code edits.
1:04:19We're all struggling with that.
1:04:21but it was like this really nice mental model of, oh yeah, we're
1:04:24gonna plan it out together.
1:04:26We're gonna design jam, how this thing's gonna work and then we're
1:04:29gonna go build it and I can just let it run and see how it ends up working.
1:04:34and Warp is doing something similar but not within the confines of VS code.
1:04:39And also with the addition of a voice button that I've been using a lot more
1:04:43recently because you can talk faster than you can type is generally what I found.
1:04:48So I can just speak into my terminal, here's how I want the app to
1:04:52work, this, that, and the other.
1:04:54And then it will, depending on how complicated the question is, it
1:04:58will reach for the planning step.
1:05:00Otherwise it'll just give you an answer right away.
1:05:01So if it, see that's kind of complicated, let me plan it out.
1:05:04It'll give you that same kind of like document, here's how it's going to work.
1:05:08And then you can say, okay, do it.
1:05:10And then it will go off and tell Claude to make file edits and do other
1:05:15things on your machine, which is much further than Warp has gone in the past.
1:05:19and I've really enjoyed using this to build Swift apps recently.
1:05:23Since I was just fascinated with like, how could I build a really
1:05:27slick desktop client like ChatGPT there's this desktop shortcut to
1:05:31like pull up a little chat bar.
1:05:33Like I want something that integrated.
1:05:36I don't wanna be confined to Google Chrome anymore.
1:05:38I wanna break out of it.
1:05:40but if you open up XCode, you're just met with this like decade old auto
1:05:45complete that doesn't have anything that you want in order to get stuff done.
1:05:50but Warp is just a terminal.
1:05:51So I'm like, okay, I'll just open the Swift project in Warp
1:05:54and say implement this feature.
1:05:56And it doesn't it, it can literally just enter any directory that you
1:06:00have and just start doing things.
1:06:02I've also used it to like migrate my open source projects from a mono repo to a
1:06:07set of micro repos on my system and says, oh yeah, I'll just make a new directory.
1:06:12I'll move all those files over and I'll make the necessary file edits with Cloud.
1:06:17Very like hit or miss quality.
1:06:19We're dialing it in.
1:06:20But this idea of you're not constrained to the IDE anymore, you can kind
1:06:25of just pull up your terminal and ask it to modify anything from the
1:06:30simplest request of, help me get revert, whatever the heck I just did.
1:06:34And it'll help you get to something more complicated, like, why
1:06:38isn't my Postgres server running?
1:06:40And then it'll check your Homebrew installation.
1:06:43And then you can take it one step further to, I actually want to fix this
1:06:47error I see in my dev server right now.
1:06:50Because you're running the dev server in your terminal, it can say, all
1:06:53right, pause the server, debug, debug, debug, restart the server.
1:06:57And then if something fails again, I'll go back to debugging.
1:07:01So it's like watching your terminal session and figuring
1:07:04out how to help you do something.
1:07:06It feels like it's this natural next step of let's go from you're in an
1:07:10editor typing code quickly to like, this is a general purpose tool on
1:07:14your machine to edit all the software that you're writing in any setting.
1:07:19so I'm just very excited about that kind of future.
1:07:22And we've been moving very, very quickly towards it.
1:07:25Just in the past month, it's gone from barely usable to, I'm actually using
1:07:30this a lot for projects in a language that I don't even know how to speak.
1:07:34Swift.
1:07:35it's been kind of crazy how far you can get from zero to one
1:07:39without a lot of field knowledge.
1:07:41And intuitively this makes a lot of sense since like Eternal is kind of like the
1:07:46OG chat up in a way where like all the way back to like IRC, et cetera, but
1:07:51also now with, using ChatGPT a lot or, or other LLM chat products, like yes,
1:07:57you're, chatting with the thing, like you write a command, the command happens
1:08:01to be like plain English or another language, and you get something back.
1:08:06But the, kinda like back and forth.
1:08:08And the interplay is very similar and it makes so much more, so much sense that you
1:08:14now bring that into the terminal as well, where you get the best of both worlds.
1:08:19You can like ride out things in a fuzzy way.
1:08:22The terminal helps you to like put that into proper computer speak.
1:08:27but then you also get the efficiency and the correctness from what you can do in
1:08:32a terminal and with all of like just a top-notch craft that you get within Warp
1:08:38as a terminal with like blocks, et cetera.
1:08:41So yeah, highly recommend everyone to, give it a try.
1:08:44Yeah.
1:08:44And it's free to reach for all those things.
1:08:47And anyone who is just bothered by AI in their terminal, you can turn it
1:08:50off and just use Warp as a really good terminal, which is how I started using
1:08:55it way back, probably like, 2021, I think is when it said I created my account.
1:09:01I just used it because I wanted something that looked nice and
1:09:04now it's going a lot deeper.
1:09:06And yeah, the, chat app analogy is perfect.
1:09:09There's literally a toggle between typing out commands and asking it a question,
1:09:14and it'll even flip back and forth based on like natural language, which is fancy.
1:09:19I mean, I'll just hit the keyboard shortcut, but why
1:09:22not make it a little flashier?
Outro
1:09:24That is awesome.
1:09:25Well, I've already seen, a bunch of your recent videos,
1:09:28about content related to that.
1:09:30I'm looking forward to many more of those.
1:09:32Is there anything else that you wanna share related to your local-first,
1:09:38explorations or otherwise?
1:09:40Yeah.
1:09:41so my profile is bholmesdev everywhere.
1:09:44So any learning resources I've put out like videos on local-first and
1:09:50conference talks, bholmesdev on YouTube and Twitter and Bluesky.
1:09:55And also on GitHub, so these Simple Sync Engine project I mentioned,
1:09:59that's on my personal GitHub.
1:10:00Also under bholmesdev.
1:10:02You should see it as one of the star repos if you look up the profile.
1:10:06That's it.
1:10:06Perfect.
1:10:07I've also put it in the show notes, so everything, you'll find it there as well.
1:10:11But, I'm really, really excited you have put in the effort to create this project,
1:10:17because I think there's no better way to learn something than trying to do it.
1:10:22And you've done that and you've allowed other people to follow
1:10:25your footsteps, and I think that's a fantastic learning resource.
1:10:29So thank you so much for doing that and for, yeah, helping others learn
1:10:34and, sharing what you've learned and for coming on the show today.
1:10:37Thank you.
1:10:38Yeah, thanks so much.
1:10:39This was like a really far reaching conversation.
1:10:42I hope it turns out good.
1:10:43Thank you for listening to the localfirst.fm podcast.
1:10:46If you've enjoyed this episode and haven't done so already, please
1:10:49subscribe and leave a review.
1:10:51Please also share this episode with your friends and colleagues.
1:10:54Spreading the word about the podcast is a great way to support
1:10:57it and to help me keep it going.
1:10:59A special thanks again to Jazz for supporting this podcast.