Lease by Physical Partition Key Range

[hallipr]

None of the documentation for change feeds mention physical partitions, but those are the leased units. I don't believe any of my collections have more that 1 physical partition, so none of our feeds are load balanced because one host gets the only available lease.

DocDBTrace Information: 0 : Source collection: 'dbs/ReleaseService/colls/Logs', 1 partition(s)
DocDBTrace Information: 0 : Host '1787-4' 1 partitions, 1 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 1, will try to take 0 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-7' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-2' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-9' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-6' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-3' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-1' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-0' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-5' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-8' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.

With 2M documents in a collection and throughput at about 150 documents / second, it would take 4 hours for the single active host to process the backlog, and that's with a no-op onChangesDelegate.

Are my collections odd in that they have only one physical?

It would seem like a lot of people assume that change feeds will load balance their partitions given that none of the documentation on change feeds indicates otherwise, but in practice, only one instance processes the entire collection.

[ealsur]

This is a fair point, and something we want to solve. We are not just there yet but we want to allow a higher degree of parallelism that is not based off physical partitions.

There are ways to increase physical partitions (more data, higher RU/s) but our current library has the REST API limitation, which is, we can only call the REST API at the physical partition, so that limits the degree of parallelism.

If what you want is a higher throughput processing the changes, what you can do is use the Change Feed as input, and fan out the operations into multiple consumer instances, and use manual checkpointing to confirm once all those consumers have finalized.

[hallipr]

We're effectively doing this, but we're still stuck behind the Change Feed as the bottleneck. Do you have baseline performance numbers for change feeds given allocated RUs? i.e. Assuming default batch size of 100 and no work performed in the onChanges method, how many message per second can a change feed processor process for 1K, 10K, 100K collections?

[ealsur]

The speed at which the changes are delivered to the Observer are basically the speed at which you can process them.

The cycle is:

1. Processor reads changes from the Change Feed
2. If there are no new changes, it sleeps for a configurable amount of time and goes back to 1.
3. If there are new changes, it sends them to your delegate.
4. Your Observer delegate receives them at processes them
5. Processor checkpoints and goes back to 1.

The rate is defined by your own implementation.

[bartelink]

and use manual checkpointing to confirm once all those consumers have finalized.

wondering if this is a reference to CheckpointAsync, or some other construct?