Sharded Databases
Sharding is often used in backend databases to handle higher data volumes.
In the case of Postgres, PowerSync cannot replicate Postgres foreign tables.
However, PowerSync does have options available to support sharded databases in general.
When using MongoDB or MySQL as the backend source database, PowerSync does not currently support connecting to sharded clusters.
The primary options are:
- Use a separate PowerSync Service instance per database.
- Add a connection for each database in the same PowerSync Service instance (planned; this capability will be available in a future release).
Where feasible, using separate PowerSync Service instances would give better performance and give more control over how changes are rolled out, especially around Sync Rule reprocessing.
Some specific scenarios:
1. Different tables on different databases
This is common when separate “services” use separate databases, but multiple tables across those databases need to be synchronized to the same users.
Use a single PowerSync Service instance, with a separate connection for each source database (planned; this capability will be available in a future release). Use a unique connection tag for each source database, allowing them to be distinguished in the Sync Rules.
2a. All data for any single customer is contained in a single shard
This is common when sharding per customer account / organization.
In this case, use a separate PowerSync Service instance for each database.
2b. Most customer data is in a single shard, but some data is in a shared database
If the amount of shared data is small, still use a separate PowerSync Service instance for each database, but also add the shared database connection to each PowerSync Service instance using a separate connection tag (planned; this capability will be available in a future release).
3. Only some tables are sharded
In some cases, most tables would be on a shared server, with only a few large tables being sharded.
For this case, use a single PowerSync Service instance. Add each shard as a new connection on this instance (planned; this capability will be available in a future release) — all with the same connection tag, so that the same Sync Rules apply to each.
Sharded Databases
Sharding is often used in backend databases to handle higher data volumes.
In the case of Postgres, PowerSync cannot replicate Postgres foreign tables.
However, PowerSync does have options available to support sharded databases in general.
When using MongoDB or MySQL as the backend source database, PowerSync does not currently support connecting to sharded clusters.
The primary options are:
- Use a separate PowerSync Service instance per database.
- Add a connection for each database in the same PowerSync Service instance (planned; this capability will be available in a future release).
Where feasible, using separate PowerSync Service instances would give better performance and give more control over how changes are rolled out, especially around Sync Rule reprocessing.
Some specific scenarios:
1. Different tables on different databases
This is common when separate “services” use separate databases, but multiple tables across those databases need to be synchronized to the same users.
Use a single PowerSync Service instance, with a separate connection for each source database (planned; this capability will be available in a future release). Use a unique connection tag for each source database, allowing them to be distinguished in the Sync Rules.
2a. All data for any single customer is contained in a single shard
This is common when sharding per customer account / organization.
In this case, use a separate PowerSync Service instance for each database.
2b. Most customer data is in a single shard, but some data is in a shared database
If the amount of shared data is small, still use a separate PowerSync Service instance for each database, but also add the shared database connection to each PowerSync Service instance using a separate connection tag (planned; this capability will be available in a future release).
3. Only some tables are sharded
In some cases, most tables would be on a shared server, with only a few large tables being sharded.
For this case, use a single PowerSync Service instance. Add each shard as a new connection on this instance (planned; this capability will be available in a future release) — all with the same connection tag, so that the same Sync Rules apply to each.