Multiple data sources

[joneslloyd]

Is it possible to have multiple data sources in Payload CMS?

By this I mean (for example) :

• A Postgres database for my Users, Pages and Media collections
• Using something like S3 storage for other (very large and less frequently updated) collections like ScrapedPages

?

[matthijs166]

It is possible to upload JSON files in the upload field.
You can make a collection with pages where an upload field is configured and upload the JSON data as file.

The S3 storage plugin would handle the upload to S3

[joneslloyd]

Sorry, I probably didn't phrase my question clearly enough.

I am aware that I can upload JSON (and other arbitrary) files, but I am asking something different.

In the Config we specify a data source, e.g.:

  db: postgresAdapter({
    pool: {
      connectionString: dbConnectionString,
      ssl: sslConfig,
    },
    prodMigrations: migrations,
  }),

I am asking whether it is possible to specify multiple data sources depending on the collection, and it's all handled programatically (like when using one data source).

I have a feeling that this is not possible based on my understanding of Payload, but I'd like to know for sure.

I'd also like to know whether this is something that could be supported in future.

[matthijs166]

Hmm don't think this is supported out of the box.
Also, I'm questing a bit why you would/need this and what problem you're tying to solve.

It is possible however to implement it yourself.
You can add some hooks to the collection to read, write and update to your other storage solution.
After committing, you can cancel the save to the default storage adapter.

Also, possible but a bit harder is to implement your own database adapter.

In the example you give, it's the best way to just use S3 and the cloud storage plugin.

[bratvanov]

I've been wanting to work with external document data in Payload for a while. My use case involves integrating a third-party service: fetching external data into the admin UI, editing it, and syncing changes back, all without storing anything in Payload’s local DB.

Right now, this is partially doable using hooks, as @matthijs166 mentioned — but only in the edit view for single documents. The list view (table of all documents) doesn't support this, since collection hooks only apply to single-document operations.

Additionally, the DefaultListView component props don’t expose document data, queries, or pagination handlers. These aren’t props yet, so you can’t simply reuse the default list component with an external data source.

I’ve opened a related feature request, #12702, proposing either support for external data via new hooks or by exposing the necessary props in DefaultListView. The request isn’t identical to this one, but there’s a lot of overlap.

[matthijs166]

Hmmm I understand you need some control over the collection overview.
Think because you don't want to use the payload layer it is a bit odd to use the overview UI for this.
The overview UI is based on the payload layer.

In this case, I would say that you need to implement your own dashboard view
https://payloadcms.com/docs/custom-components/custom-views#adding-new-views
And in this view, you can use the @payloadcms/ui package to build your own UI implementation and implement your own endpoints with server logic to external fetch the data.

[rtlayzell]

I have the same need for something like this. We have an existing database for an ECommerce site that has complex filtering logic for the products, built into our own API's. The product catalogue and variants together makes up around 100,000 unique products and the catalogue is synced via an automated process. It's not feasible to simply add all the products to payload manually nor is the client API flexible enough to handle the filter logic.
What we would like to be able to do is get visibility from the admin UI and tweak the collection/fields, if and when necessary (to start). I've tried the hooks but as the comment above suggests, it's only applicable to individual documents. I need a way to completely replace payload's CRUD operations for specific collections.