I think what we're seeing with every database company providing new full-text search capabilities is an example of AI coding productivity showing up in the real world.
AFAIK all of these are implementations of the BM25 algorithm. You can just tell an agent to read about BM25 and implement it in your system of choice. Cool to see. Seems like there's still a lot of juice to be squeezed out of how it's architected and integrated into each system, but you can't help but wonder if this will lead to aggressive commodification
There is a lot of truth to this, but it's also very much down to domain experts being able to do this to move faster.
Planetscale (assuming they used a agentic development practice) will have pulled this off, to the level of performance that they have, because they have a team of very highly experienced Postgres developers. Their knowlage of Postgres internals will have given them the insights needed to steer the models to a plan that used the architecture as described in the post. That's not something a model can do on its own*
World experts + LLMs = moving mountains.
(* we're obviously seeing something a little different from inside the research teams in the labs. They are showing that the models, when you burn the level of tokens only they can, are able to do novel things from the models own insights.)
Becoming more the norm for them, Neki is the same.
Immediately rules out ever using them (though I don't currently have any problems that would benefit from that level of scale currently, have in the past though).
Postgres's license allows this but for me (personally) it leaves a bad taste.
Also it's not really "full-text search for Postgres" it's "full-text search for our hosted version of Postgres" so the title is a little misleading.
Building non-open extensions on top of it, it's not the license I don't like, the bad taste is that they use something open extend it and keep part of it closed.
The license allows it but on the flip side it's vendor lock-in predicated on using something open as the base.
Fully proprietary no issue with that, full open, no issue with that, building proprietary on top of open is where the bad taste comes in.
For completeness, it's not them specifically either, the other cloud companies do similar things and I suspect in part the reason they don't open these extensions up is because the others will but then they are doing the same thing themselves.
When one develops using open-source software they have an obligation to follow the licenses. They also have a moral obligation to be respectful of the work upon which they’re building. And they have a social obligation to help improve that software where they can.
Those that develop on top of open-source have no obligation to give you their work for free.
You’d be surprised as to the amount of open-source contributions TIN drove towards Postgres, LLVM, and pgrx. And you’d be speechless at the amount of upstream work across all sorts of open-source PlanetScale does. Postgres 18.6, for example, is better for you today, in part, because of TIN. You’re welcome.
Interestingly enough SQLites FTS supports Lucene queries out of the box with great performance characteristics. IIRC only writes become pretty slow after a while.
I’ve always wondered what exactly would prevent PostgreSQL from strapping that implementation into its own database. My experience with ts_query hasn’t been particularly rosy. It can be better than LIKE but only marginally so and at the cost of insane index sizes…
If this extension becomes open source and we can test it out in the real world I’m sure there’s a sweet spot
SQLite FTS relies on shadow B-trees under single-writer locks. Postgres index access methods must map postings directly to physical ctid tuples, surviving MVCC visibility checks and heap tuple churn.
I struggle with FTS inside SQL (SQLite and MSSQL). There is often a fairly significant impedance mismatch between the relational concerns and how the documents need to be stored.
I've always preferred to use SQL as the system of record and then build/maintain an external Lucene index. Do we think these integral FTS capabilities are at the point where a hybrid architecture doesn't make sense anymore? How much customization exists in this provider?
I’m one of TIN’s developers and if you google my username you’ll see I’ve been in this space for a long time.
The answer to your first question is simply: yes
As far as your second question, what customization do you need that you believe TIN or PlanetScale doesn’t provide? These are things we can do, with alacrity.
in my experience it’s pretty common to find big inverted indexes for text directly in the database - not necessarily large docs but certainly free text records in volume. using bm25 and unicode’s breakiterator is a very good way to build it. like putting lucene in the database basically - makes a lot of sense when the database is already large. places that bend over backwards to move search out of the db are usually trying to avoid having a very large db (and often end up with one anyway, getting the worst of both worlds)
They also end up with all the infrastructure and processes necessary to keep the external search system in sync, resync/reindex, pkey shipping back to their source of truth in queries, application-side joins and enrichment between both sources. It’s brutal.
Having everything in one place eliminates entire classes of development and especially operational problems.
It’s just me or there are others who keep seeing these updates and think mongodb had all of this years ago?
Seriously so happy to be running our production stack on mongo.
I think what we're seeing with every database company providing new full-text search capabilities is an example of AI coding productivity showing up in the real world.
It started with paradeDB and pg_search https://www.paradedb.com/blog/introducing-search
Timescale has pg_textsearch https://github.com/timescale/pg_textsearch
Neon and Databricks have Lakebase Search https://docs.databricks.com/aws/en/oltp/projects/lakebase-se...
Now PlanetScale.
AFAIK all of these are implementations of the BM25 algorithm. You can just tell an agent to read about BM25 and implement it in your system of choice. Cool to see. Seems like there's still a lot of juice to be squeezed out of how it's architected and integrated into each system, but you can't help but wonder if this will lead to aggressive commodification
There is a lot of truth to this, but it's also very much down to domain experts being able to do this to move faster.
Planetscale (assuming they used a agentic development practice) will have pulled this off, to the level of performance that they have, because they have a team of very highly experienced Postgres developers. Their knowlage of Postgres internals will have given them the insights needed to steer the models to a plan that used the architecture as described in the post. That's not something a model can do on its own*
World experts + LLMs = moving mountains.
(* we're obviously seeing something a little different from inside the research teams in the labs. They are showing that the models, when you burn the level of tokens only they can, are able to do novel things from the models own insights.)
ParadeDB's implementation builds on the Tantivy crate, which predates AI coding.
If anyone's curious - https://planetscale.com/docs/postgres/search/get-started#loc...:
They're not providing a local extension with the same performance at the time - it's only offered on their cloud services.
The local version https://github.com/planetscale/lead is mainly just for testing the syntax, it doesn't have the same perf characteristics.
Becoming more the norm for them, Neki is the same.
Immediately rules out ever using them (though I don't currently have any problems that would benefit from that level of scale currently, have in the past though).
Postgres's license allows this but for me (personally) it leaves a bad taste.
Also it's not really "full-text search for Postgres" it's "full-text search for our hosted version of Postgres" so the title is a little misleading.
Why don’t you like Postgres’ license? It’s as permissive as a license gets.
Building non-open extensions on top of it, it's not the license I don't like, the bad taste is that they use something open extend it and keep part of it closed.
The license allows it but on the flip side it's vendor lock-in predicated on using something open as the base.
Fully proprietary no issue with that, full open, no issue with that, building proprietary on top of open is where the bad taste comes in.
For completeness, it's not them specifically either, the other cloud companies do similar things and I suspect in part the reason they don't open these extensions up is because the others will but then they are doing the same thing themselves.
When one develops using open-source software they have an obligation to follow the licenses. They also have a moral obligation to be respectful of the work upon which they’re building. And they have a social obligation to help improve that software where they can.
Those that develop on top of open-source have no obligation to give you their work for free.
You’d be surprised as to the amount of open-source contributions TIN drove towards Postgres, LLVM, and pgrx. And you’d be speechless at the amount of upstream work across all sorts of open-source PlanetScale does. Postgres 18.6, for example, is better for you today, in part, because of TIN. You’re welcome.
Why would you need super fast search for local testing?
To avoid vendor lock-in.
It's Postgres search any coding agent can switch you to something else in 5 minutes
why?
The problem is that they don’t support bare metal. I’d love to use PlanetScale in our bare metal servers.
we support bare metal inside AWS, GCP, and very soon Azure
Interestingly enough SQLites FTS supports Lucene queries out of the box with great performance characteristics. IIRC only writes become pretty slow after a while. I’ve always wondered what exactly would prevent PostgreSQL from strapping that implementation into its own database. My experience with ts_query hasn’t been particularly rosy. It can be better than LIKE but only marginally so and at the cost of insane index sizes… If this extension becomes open source and we can test it out in the real world I’m sure there’s a sweet spot
SQLite FTS relies on shadow B-trees under single-writer locks. Postgres index access methods must map postings directly to physical ctid tuples, surviving MVCC visibility checks and heap tuple churn.
I struggle with FTS inside SQL (SQLite and MSSQL). There is often a fairly significant impedance mismatch between the relational concerns and how the documents need to be stored.
I've always preferred to use SQL as the system of record and then build/maintain an external Lucene index. Do we think these integral FTS capabilities are at the point where a hybrid architecture doesn't make sense anymore? How much customization exists in this provider?
I’m one of TIN’s developers and if you google my username you’ll see I’ve been in this space for a long time.
The answer to your first question is simply: yes
As far as your second question, what customization do you need that you believe TIN or PlanetScale doesn’t provide? These are things we can do, with alacrity.
in my experience it’s pretty common to find big inverted indexes for text directly in the database - not necessarily large docs but certainly free text records in volume. using bm25 and unicode’s breakiterator is a very good way to build it. like putting lucene in the database basically - makes a lot of sense when the database is already large. places that bend over backwards to move search out of the db are usually trying to avoid having a very large db (and often end up with one anyway, getting the worst of both worlds)
They also end up with all the infrastructure and processes necessary to keep the external search system in sync, resync/reindex, pkey shipping back to their source of truth in queries, application-side joins and enrichment between both sources. It’s brutal.
Having everything in one place eliminates entire classes of development and especially operational problems.
What’s funny is I worked with a company with planet in the name Who could really use a full text search that was great in the Postgres