2026-04-14 14:40:02 | Log4Net | Unhandled exception in DiscardArchivalMaterials: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 09e79054-5e9e-4e32-8489-b7122b3e0eb5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 09e79054-5e9e-4e32-8489-b7122b3e0eb5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 09e79054-5e9e-4e32-8489-b7122b3e0eb5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 09e79054-5e9e-4e32-8489-b7122b3e0eb5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T14:40:02.694Z","duration in milliseconds":125.26150000000001,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T14:39:59.1470285Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":89928,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0475},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":125.0229,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":124.9766,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T14:39:07.2342069Z","cpu":10.859,"memory":5278816.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1101,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:39:17.2458627Z","cpu":4.914,"memory":5273392.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1246,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":4},{"dateUtc":"2026-04-14T14:39:27.2606224Z","cpu":4.992,"memory":5280288.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.092,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:39:37.2765257Z","cpu":2.184,"memory":5279648.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1137,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1},{"dateUtc":"2026-04-14T14:39:47.2909857Z","cpu":8.190,"memory":5264152.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0586,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":2},{"dateUtc":"2026-04-14T14:39:57.3023312Z","cpu":6.406,"memory":5263268.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0614,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":124.96390000000001,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":124.95880000000001,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":124.2686,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":124.2611,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":115.85170000000001,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T14:40:02.7095971Z","DurationInMs":15.5584,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"09e79054-5e9e-4e32-8489-b7122b3e0eb5","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26148441,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26148441,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26148441,"SessionToken":"-1#26148441","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.858","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 2:40:00 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T14:40:02.6940387Z", "durationInMs": 0.004},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T14:40:02.6940427Z", "durationInMs": 0.0049},{"event": "Pipelined", "startTimeUtc": "2026-04-14T14:40:02.6940476Z", "durationInMs": 0.2298},{"event": "Transit Time", "startTimeUtc": "2026-04-14T14:40:02.6942774Z", "durationInMs": 2.1001},{"event": "Received", "startTimeUtc": "2026-04-14T14:40:02.6963775Z", "durationInMs": 0.0723},{"event": "Completed", "startTimeUtc": "2026-04-14T14:40:02.6964498Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T14:40:02.6783501Z","lastSend":"2026-04-14T14:40:02.6783501Z","lastReceive":"2026-04-14T14:40:02.6940387Z"},"requestSizeInBytes":1561,"requestBodySizeInBytes":1107,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":3.6564126,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"09e79054-5e9e-4e32-8489-b7122b3e0eb5","ResponseTimeUtc":"2026-04-14T14:40:02.8189699Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 09e79054-5e9e-4e32-8489-b7122b3e0eb5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26148441","BELatencyInMs":"0.858"}}}]}]}]}]}]}]}]}
2026-04-14 14:40:30 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: deaa5e0a-f28e-4635-9ee6-f5a1775a2fb8; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: deaa5e0a-f28e-4635-9ee6-f5a1775a2fb8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: deaa5e0a-f28e-4635-9ee6-f5a1775a2fb8; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: deaa5e0a-f28e-4635-9ee6-f5a1775a2fb8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T14:40:30.717Z","duration in milliseconds":56.922200000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T14:40:29.9984165Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":89931,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.043000000000000003},{"name":"Get PkValue From Stream","duration in milliseconds":7.077,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0035},{"name":"Read Collection","duration in milliseconds":6.9291,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T14:40:30.7171765Z","DurationInMs":15.692900000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"b0a90a8c-ffe5-482e-83f4-4865c6f93468","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":40.8331,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":40.8031,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T14:39:37.2765257Z","cpu":2.184,"memory":5279648.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1137,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1},{"dateUtc":"2026-04-14T14:39:47.2909857Z","cpu":8.190,"memory":5264152.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0586,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":2},{"dateUtc":"2026-04-14T14:39:57.3023312Z","cpu":6.406,"memory":5263268.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0614,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:40:07.3094632Z","cpu":7.254,"memory":5249284.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1154,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:40:17.3212123Z","cpu":2.964,"memory":5240308.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2039,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:40:27.3232594Z","cpu":0.859,"memory":5251172.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1007,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":40.7948,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":40.7907,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":40.7701,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":40.7661,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":39.8953,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T14:40:30.7328694Z","EndTimeUTC":"2026-04-14T14:40:30.7484290Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T14:40:30.7641053Z","DurationInMs":15.676300000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"deaa5e0a-f28e-4635-9ee6-f5a1775a2fb8","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26148441,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26148441,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26148441,"SessionToken":"-1#26148441","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.658","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 2:40:30 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T14:40:30.7484290Z", "durationInMs": 0.0095},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T14:40:30.7484385Z", "durationInMs": 23.2309},{"event": "Pipelined", "startTimeUtc": "2026-04-14T14:40:30.7716694Z", "durationInMs": 0.2053},{"event": "Transit Time", "startTimeUtc": "2026-04-14T14:40:30.7718747Z", "durationInMs": 1.827},{"event": "Received", "startTimeUtc": "2026-04-14T14:40:30.7737017Z", "durationInMs": 0.1448},{"event": "Completed", "startTimeUtc": "2026-04-14T14:40:30.7738465Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T14:40:30.7641053Z","lastSend":"2026-04-14T14:40:30.7641053Z","lastReceive":"2026-04-14T14:40:30.7641053Z"},"requestSizeInBytes":1628,"requestBodySizeInBytes":1178,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T14:40:30.7328694Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"c4a06251-8d0b-4d1f-babd-4cdcf8b8de4a","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T14:40:30.7328694Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"b405c0f4-58f9-41f4-8a96-e27f9d65fd65","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T14:40:30.7328694Z","DurationInMs":15.559600000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"deaa5e0a-f28e-4635-9ee6-f5a1775a2fb8","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.7656888,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"deaa5e0a-f28e-4635-9ee6-f5a1775a2fb8","ResponseTimeUtc":"2026-04-14T14:40:30.7641053Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: deaa5e0a-f28e-4635-9ee6-f5a1775a2fb8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26148441","BELatencyInMs":"0.658"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 14:47:52 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c704e198-ae4d-428e-9287-112f0fad9c7a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c704e198-ae4d-428e-9287-112f0fad9c7a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c704e198-ae4d-428e-9287-112f0fad9c7a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c704e198-ae4d-428e-9287-112f0fad9c7a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T14:47:52.372Z","duration in milliseconds":31.910200000000003,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T14:47:51.9664187Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90020,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.047400000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":0.1898,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0016}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":31.264000000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":31.2188,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T14:46:57.6727893Z","cpu":1.872,"memory":5182444.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0705,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:47:07.6758820Z","cpu":1.406,"memory":5181836.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0949,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:47:17.6834665Z","cpu":2.422,"memory":5182084.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3021,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:47:27.6980911Z","cpu":0.858,"memory":5185924.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.119,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:47:37.7070689Z","cpu":4.680,"memory":5151268.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.109,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:47:47.7166423Z","cpu":1.172,"memory":5153308.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.4157,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":31.1932,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":31.183500000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":31.153200000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":31.146900000000002,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":29.6685,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T14:47:52.3882981Z","DurationInMs":15.629100000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c704e198-ae4d-428e-9287-112f0fad9c7a","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26148552,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26148552,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26148552,"SessionToken":"-1#26148552","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.955","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 2:47:52 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T14:47:52.3726690Z", "durationInMs": 0.0184},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T14:47:52.3726874Z", "durationInMs": 14.5123},{"event": "Pipelined", "startTimeUtc": "2026-04-14T14:47:52.3871997Z", "durationInMs": 0.2333},{"event": "Transit Time", "startTimeUtc": "2026-04-14T14:47:52.3874330Z", "durationInMs": 3.4982},{"event": "Received", "startTimeUtc": "2026-04-14T14:47:52.3909312Z", "durationInMs": 0.1716},{"event": "Completed", "startTimeUtc": "2026-04-14T14:47:52.3911028Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T14:47:52.3882981Z","lastSend":"2026-04-14T14:47:52.3882981Z","lastReceive":"2026-04-14T14:47:52.3882981Z"},"requestSizeInBytes":1637,"requestBodySizeInBytes":1183,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.4375032,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c704e198-ae4d-428e-9287-112f0fad9c7a","ResponseTimeUtc":"2026-04-14T14:47:52.4039219Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c704e198-ae4d-428e-9287-112f0fad9c7a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26148552","BELatencyInMs":"0.955"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 14:49:54 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 7fb8afa6-0bbb-4631-8fb2-a0d583d453e7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 7fb8afa6-0bbb-4631-8fb2-a0d583d453e7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 7fb8afa6-0bbb-4631-8fb2-a0d583d453e7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 7fb8afa6-0bbb-4631-8fb2-a0d583d453e7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T14:49:54.314Z","duration in milliseconds":42.8814,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T14:49:54.2209977Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90070,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0509},{"name":"Get PkValue From Stream","duration in milliseconds":7.5494,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0016},{"name":"Read Collection","duration in milliseconds":7.3751000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T14:49:54.3148265Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"476be4e4-b47c-4ba7-869a-2cc9713ba649","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.097100000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.053200000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T14:48:57.7867240Z","cpu":5.781,"memory":5317440.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1061,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:49:07.8028155Z","cpu":2.106,"memory":5304316.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0969,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:49:17.8045355Z","cpu":0.703,"memory":5301800.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1038,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:49:27.8080891Z","cpu":6.797,"memory":5292488.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.108,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:49:37.8175693Z","cpu":9.282,"memory":5289116.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1576,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:49:47.8189714Z","cpu":7.266,"memory":5269392.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0955,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.0458,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.04,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.0172,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.0145,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.013200000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T14:49:54.3303757Z","EndTimeUTC":"2026-04-14T14:49:54.3303757Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T14:49:54.3460095Z","DurationInMs":15.6338,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"7fb8afa6-0bbb-4631-8fb2-a0d583d453e7","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26148553,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26148553,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26148553,"SessionToken":"-1#26148553","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.613","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 2:49:54 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T14:49:54.3303757Z", "durationInMs": 0.0083},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T14:49:54.3303840Z", "durationInMs": 18.1161},{"event": "Pipelined", "startTimeUtc": "2026-04-14T14:49:54.3485001Z", "durationInMs": 0.2012},{"event": "Transit Time", "startTimeUtc": "2026-04-14T14:49:54.3487013Z", "durationInMs": 1.9608},{"event": "Received", "startTimeUtc": "2026-04-14T14:49:54.3506621Z", "durationInMs": 0.1427},{"event": "Completed", "startTimeUtc": "2026-04-14T14:49:54.3508048Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T14:49:54.3460095Z","lastSend":"2026-04-14T14:49:54.3460095Z","lastReceive":"2026-04-14T14:49:54.3460095Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T14:49:54.3148265Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"9affe174-5afe-455b-98a3-f54da4dfd52c","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T14:49:54.3148265Z","DurationInMs":15.5492,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"9e32d110-7d1e-4ceb-addb-6e4bedce064c","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T14:49:54.3303757Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"7fb8afa6-0bbb-4631-8fb2-a0d583d453e7","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1250118,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"7fb8afa6-0bbb-4631-8fb2-a0d583d453e7","ResponseTimeUtc":"2026-04-14T14:49:54.3460095Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 7fb8afa6-0bbb-4631-8fb2-a0d583d453e7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26148553","BELatencyInMs":"0.613"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001}]}
2026-04-14 14:50:20 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 3749c6ad-c0c5-46e4-adb1-ff6fed4d6ec5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 3749c6ad-c0c5-46e4-adb1-ff6fed4d6ec5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 3749c6ad-c0c5-46e4-adb1-ff6fed4d6ec5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 3749c6ad-c0c5-46e4-adb1-ff6fed4d6ec5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T14:50:20.511Z","duration in milliseconds":46.9649,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T14:50:20.4019808Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90085,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.051800000000000006},{"name":"Get PkValue From Stream","duration in milliseconds":7.1908,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0022},{"name":"Read Collection","duration in milliseconds":7.0318000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T14:50:20.5113588Z","DurationInMs":15.679300000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"7fe51019-3b1d-4d68-8658-697711b2eb89","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":39.5379,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":39.504200000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T14:49:27.8080891Z","cpu":6.797,"memory":5292488.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.108,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:49:37.8175693Z","cpu":9.282,"memory":5289116.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1576,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:49:47.8189714Z","cpu":7.266,"memory":5269392.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0955,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:49:57.8252990Z","cpu":5.000,"memory":5269104.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1113,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:50:07.8356546Z","cpu":18.487,"memory":5138608.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1209,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:50:17.8395072Z","cpu":4.531,"memory":5161060.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1231,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":39.495000000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":39.4909,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":39.4694,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":39.466100000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":38.574000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T14:50:20.5270381Z","EndTimeUTC":"2026-04-14T14:50:20.5426164Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T14:50:20.5583080Z","DurationInMs":15.691600000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"3749c6ad-c0c5-46e4-adb1-ff6fed4d6ec5","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26148577,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26148577,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26148577,"SessionToken":"-1#26148577","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.679","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 2:50:20 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T14:50:20.5426164Z", "durationInMs": 0.0092},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T14:50:20.5426256Z", "durationInMs": 21.8583},{"event": "Pipelined", "startTimeUtc": "2026-04-14T14:50:20.5644839Z", "durationInMs": 0.1893},{"event": "Transit Time", "startTimeUtc": "2026-04-14T14:50:20.5646732Z", "durationInMs": 2.3455},{"event": "Received", "startTimeUtc": "2026-04-14T14:50:20.5670187Z", "durationInMs": 0.1544},{"event": "Completed", "startTimeUtc": "2026-04-14T14:50:20.5671731Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T14:50:20.5583080Z","lastSend":"2026-04-14T14:50:20.5583080Z","lastReceive":"2026-04-14T14:50:20.5583080Z"},"requestSizeInBytes":1630,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T14:50:20.5270381Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"106e3bc9-09d0-46be-a756-a31e59bdb5b7","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T14:50:20.5270381Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"61d616b0-c114-4b69-a605-8c8d233b5147","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T14:50:20.5270381Z","DurationInMs":15.5783,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"3749c6ad-c0c5-46e4-adb1-ff6fed4d6ec5","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1563272,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"3749c6ad-c0c5-46e4-adb1-ff6fed4d6ec5","ResponseTimeUtc":"2026-04-14T14:50:20.5583080Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 3749c6ad-c0c5-46e4-adb1-ff6fed4d6ec5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26148577","BELatencyInMs":"0.679"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0029000000000000002}]}
2026-04-14 14:50:50 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 59c247ef-496e-4965-967f-0a1dd06718ba; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 59c247ef-496e-4965-967f-0a1dd06718ba, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 59c247ef-496e-4965-967f-0a1dd06718ba; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 59c247ef-496e-4965-967f-0a1dd06718ba, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T14:50:50.749Z","duration in milliseconds":50.8185,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T14:50:50.6249719Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90094,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0369},{"name":"Get PkValue From Stream","duration in milliseconds":7.99,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.001},{"name":"Read Collection","duration in milliseconds":7.8061000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T14:50:50.7499764Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"ca42defc-7f70-4367-be7e-0fd5d90cd387","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":42.6147,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":42.5762,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T14:49:57.8252990Z","cpu":5.000,"memory":5269104.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1113,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:50:07.8356546Z","cpu":18.487,"memory":5138608.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1209,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:50:17.8395072Z","cpu":4.531,"memory":5161060.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1231,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:50:27.8432061Z","cpu":3.750,"memory":5158640.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1265,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:50:37.8498955Z","cpu":2.418,"memory":5198228.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1211,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:50:47.8550986Z","cpu":1.484,"memory":5195520.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0947,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":42.5687,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":42.5644,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":42.5452,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":42.542,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":41.571000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T14:50:50.7656028Z","EndTimeUTC":"2026-04-14T14:50:50.7656028Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T14:50:50.7968542Z","DurationInMs":31.2514,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"59c247ef-496e-4965-967f-0a1dd06718ba","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26148593,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26148593,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26148593,"SessionToken":"-1#26148593","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.656","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 2:50:50 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T14:50:50.7656028Z", "durationInMs": 0.0098},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T14:50:50.7656126Z", "durationInMs": 21.7322},{"event": "Pipelined", "startTimeUtc": "2026-04-14T14:50:50.7873448Z", "durationInMs": 0.2363},{"event": "Transit Time", "startTimeUtc": "2026-04-14T14:50:50.7875811Z", "durationInMs": 2.4045},{"event": "Received", "startTimeUtc": "2026-04-14T14:50:50.7899856Z", "durationInMs": 0.1317},{"event": "Completed", "startTimeUtc": "2026-04-14T14:50:50.7901173Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T14:50:50.7812267Z","lastSend":"2026-04-14T14:50:50.7812267Z","lastReceive":"2026-04-14T14:50:50.7812267Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T14:50:50.7499764Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"87dfda46-950a-42f8-b25d-9d57b99e6e0e","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T14:50:50.7499764Z","DurationInMs":15.6264,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"435a519f-de75-42f3-bf48-4fc09e3e5931","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T14:50:50.7656028Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"59c247ef-496e-4965-967f-0a1dd06718ba","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718823,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"59c247ef-496e-4965-967f-0a1dd06718ba","ResponseTimeUtc":"2026-04-14T14:50:50.7968542Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 59c247ef-496e-4965-967f-0a1dd06718ba, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26148593","BELatencyInMs":"0.656"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0014}]}
2026-04-14 14:52:43 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 7ac31f47-1742-4000-b9e3-477a73d71cad; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 7ac31f47-1742-4000-b9e3-477a73d71cad, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 7ac31f47-1742-4000-b9e3-477a73d71cad; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 7ac31f47-1742-4000-b9e3-477a73d71cad, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T14:52:43.571Z","duration in milliseconds":67.026800000000009,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T14:52:43.4303967Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90113,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0427},{"name":"Get PkValue From Stream","duration in milliseconds":7.9552000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0016},{"name":"Read Collection","duration in milliseconds":7.7963000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T14:52:43.5710203Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"4624086f-8fa8-4c1f-b77c-0a78932769a7","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":58.8547,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":58.821200000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T14:51:47.9031369Z","cpu":1.404,"memory":5193048.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0967,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:51:57.9055477Z","cpu":1.563,"memory":5189584.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1053,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:52:07.9100948Z","cpu":6.328,"memory":5124936.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0891,"availableThreads":8187,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:52:17.9119022Z","cpu":0.780,"memory":5129096.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1391,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:52:27.9247372Z","cpu":1.328,"memory":5146296.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0883,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:52:37.9319937Z","cpu":1.872,"memory":5141440.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0962,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":58.812200000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":58.8082,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":58.7868,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":58.7828,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":57.6871,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T14:52:43.5866454Z","EndTimeUTC":"2026-04-14T14:52:43.5866454Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T14:52:43.6335205Z","DurationInMs":46.8751,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"7ac31f47-1742-4000-b9e3-477a73d71cad","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26148593,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26148593,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26148593,"SessionToken":"-1#26148593","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.66","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 2:52:43 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T14:52:43.5866454Z", "durationInMs": 0.0092},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T14:52:43.5866546Z", "durationInMs": 39.1522},{"event": "Pipelined", "startTimeUtc": "2026-04-14T14:52:43.6258068Z", "durationInMs": 0.2137},{"event": "Transit Time", "startTimeUtc": "2026-04-14T14:52:43.6260205Z", "durationInMs": 1.9037},{"event": "Received", "startTimeUtc": "2026-04-14T14:52:43.6279242Z", "durationInMs": 0.1428},{"event": "Completed", "startTimeUtc": "2026-04-14T14:52:43.6280670Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T14:52:43.6335205Z","lastSend":"2026-04-14T14:52:43.6335205Z","lastReceive":"2026-04-14T14:52:43.6335205Z"},"requestSizeInBytes":1628,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T14:52:43.5710203Z","DurationInMs":15.625100000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"f0b4bba3-21ec-4e19-947e-ac106b0dd838","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T14:52:43.5866454Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"11923586-f68f-4e2e-9dbf-910c39d55f0e","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T14:52:43.5866454Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"7ac31f47-1742-4000-b9e3-477a73d71cad","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2031238,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"7ac31f47-1742-4000-b9e3-477a73d71cad","ResponseTimeUtc":"2026-04-14T14:52:43.6335205Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 7ac31f47-1742-4000-b9e3-477a73d71cad, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26148593","BELatencyInMs":"0.66"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 14:53:47 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c822861d-b010-4795-8892-49c25e93bf3b; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c822861d-b010-4795-8892-49c25e93bf3b, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c822861d-b010-4795-8892-49c25e93bf3b; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c822861d-b010-4795-8892-49c25e93bf3b, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T14:53:46.985Z","duration in milliseconds":60.1743,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T14:53:46.8763026Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90128,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0531},{"name":"Get PkValue From Stream","duration in milliseconds":7.8378000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0037},{"name":"Read Collection","duration in milliseconds":7.6622,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T14:53:46.9856839Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"39b3176b-26d0-47e6-87f8-126b9cab51a8","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":52.1075,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":52.0722,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T14:52:47.9474798Z","cpu":3.822,"memory":5135572.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0972,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:52:57.9490752Z","cpu":1.953,"memory":5134832.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0834,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:53:07.9532222Z","cpu":2.969,"memory":5282784.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1316,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:53:17.9671699Z","cpu":1.248,"memory":5282608.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1122,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:53:27.9755224Z","cpu":2.422,"memory":5278896.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.098,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:53:37.9901962Z","cpu":1.170,"memory":5278996.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1045,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":52.0629,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":52.057900000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":52.0371,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":52.0345,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":51.006800000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T14:53:47.0013083Z","EndTimeUTC":"2026-04-14T14:53:47.0013083Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T14:53:47.0481800Z","DurationInMs":46.871700000000004,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c822861d-b010-4795-8892-49c25e93bf3b","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26148593,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26148593,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26148593,"SessionToken":"-1#26148593","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.586","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 2:53:47 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T14:53:47.0013083Z", "durationInMs": 0.0089},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T14:53:47.0013172Z", "durationInMs": 33.4238},{"event": "Pipelined", "startTimeUtc": "2026-04-14T14:53:47.0347410Z", "durationInMs": 0.2088},{"event": "Transit Time", "startTimeUtc": "2026-04-14T14:53:47.0349498Z", "durationInMs": 1.9974},{"event": "Received", "startTimeUtc": "2026-04-14T14:53:47.0369472Z", "durationInMs": 0.1397},{"event": "Completed", "startTimeUtc": "2026-04-14T14:53:47.0370869Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T14:53:47.0325586Z","lastSend":"2026-04-14T14:53:47.0325586Z","lastReceive":"2026-04-14T14:53:47.0481800Z"},"requestSizeInBytes":1630,"requestBodySizeInBytes":1178,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T14:53:47.0013083Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"42464401-231f-49ef-9111-49d083437b0b","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T14:53:47.0013083Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"62e835a2-8a7f-449f-91a1-02bd2fd77e18","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T14:53:47.0013083Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"c822861d-b010-4795-8892-49c25e93bf3b","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.17187739999999999,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c822861d-b010-4795-8892-49c25e93bf3b","ResponseTimeUtc":"2026-04-14T14:53:47.0481800Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c822861d-b010-4795-8892-49c25e93bf3b, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26148593","BELatencyInMs":"0.586"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 14:53:51 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: fd483b9f-9b04-40f9-aa2a-040e0f6a3905; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: fd483b9f-9b04-40f9-aa2a-040e0f6a3905, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: fd483b9f-9b04-40f9-aa2a-040e0f6a3905; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: fd483b9f-9b04-40f9-aa2a-040e0f6a3905, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T14:53:51.161Z","duration in milliseconds":21.5518,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T14:53:50.7553585Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90130,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.051800000000000006},{"name":"Get PkValue From Stream","duration in milliseconds":0.1766,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0013000000000000002}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":21.148600000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":21.1131,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T14:52:57.9490752Z","cpu":1.953,"memory":5134832.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0834,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:53:07.9532222Z","cpu":2.969,"memory":5282784.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1316,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:53:17.9671699Z","cpu":1.248,"memory":5282608.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1122,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:53:27.9755224Z","cpu":2.422,"memory":5278896.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.098,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:53:37.9901962Z","cpu":1.170,"memory":5278996.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1045,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:53:47.9916285Z","cpu":1.719,"memory":5286480.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1261,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":21.106,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":21.101300000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":21.082700000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":21.0803,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":20.0027,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T14:53:51.1771661Z","DurationInMs":15.6257,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"fd483b9f-9b04-40f9-aa2a-040e0f6a3905","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26148593,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26148593,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26148593,"SessionToken":"-1#26148593","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.804","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 2:53:51 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T14:53:51.1615404Z", "durationInMs": 0.015},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T14:53:51.1615554Z", "durationInMs": 16.6133},{"event": "Pipelined", "startTimeUtc": "2026-04-14T14:53:51.1781687Z", "durationInMs": 0.2093},{"event": "Transit Time", "startTimeUtc": "2026-04-14T14:53:51.1783780Z", "durationInMs": 2.2132},{"event": "Received", "startTimeUtc": "2026-04-14T14:53:51.1805912Z", "durationInMs": 0.1472},{"event": "Completed", "startTimeUtc": "2026-04-14T14:53:51.1807384Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T14:53:51.1771661Z","lastSend":"2026-04-14T14:53:51.1771661Z","lastReceive":"2026-04-14T14:53:51.1771661Z"},"requestSizeInBytes":1631,"requestBodySizeInBytes":1183,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.4218076,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"fd483b9f-9b04-40f9-aa2a-040e0f6a3905","ResponseTimeUtc":"2026-04-14T14:53:51.1927927Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: fd483b9f-9b04-40f9-aa2a-040e0f6a3905, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26148593","BELatencyInMs":"0.804"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 14:58:22 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 90b8bb5b-997f-427e-909d-9747a6383c4a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 90b8bb5b-997f-427e-909d-9747a6383c4a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 90b8bb5b-997f-427e-909d-9747a6383c4a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 90b8bb5b-997f-427e-909d-9747a6383c4a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T14:58:22.160Z","duration in milliseconds":21.762800000000002,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T14:58:21.5662596Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90189,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0529},{"name":"Get PkValue From Stream","duration in milliseconds":0.15940000000000001,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":21.372400000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":21.3393,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T14:57:28.2008720Z","cpu":1.950,"memory":5215880.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1689,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:57:38.2030453Z","cpu":2.813,"memory":5215580.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0779,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:57:48.2065687Z","cpu":2.266,"memory":5221732.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.223,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:57:58.2224061Z","cpu":2.964,"memory":5295044.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0613,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:58:08.2339135Z","cpu":1.014,"memory":5280004.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1156,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T14:58:18.2469802Z","cpu":1.482,"memory":5274500.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1029,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":21.332900000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":21.3289,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":21.3093,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":21.307100000000002,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":20.3243,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T14:58:22.1757100Z","DurationInMs":15.7058,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"90b8bb5b-997f-427e-909d-9747a6383c4a","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26148636,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26148636,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26148636,"SessionToken":"-1#26148636","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.865","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 2:58:22 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T14:58:22.1600042Z", "durationInMs": 0.0138},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T14:58:22.1600180Z", "durationInMs": 16.9776},{"event": "Pipelined", "startTimeUtc": "2026-04-14T14:58:22.1769956Z", "durationInMs": 0.2191},{"event": "Transit Time", "startTimeUtc": "2026-04-14T14:58:22.1772147Z", "durationInMs": 2.2009},{"event": "Received", "startTimeUtc": "2026-04-14T14:58:22.1794156Z", "durationInMs": 0.1351},{"event": "Completed", "startTimeUtc": "2026-04-14T14:58:22.1795507Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T14:58:22.1757100Z","lastSend":"2026-04-14T14:58:22.1757100Z","lastReceive":"2026-04-14T14:58:22.1757100Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.6094504,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"90b8bb5b-997f-427e-909d-9747a6383c4a","ResponseTimeUtc":"2026-04-14T14:58:22.1757100Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 90b8bb5b-997f-427e-909d-9747a6383c4a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26148636","BELatencyInMs":"0.865"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 15:01:27 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 3b066746-ec51-4d7a-8d62-a8a18b2cbadc; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 3b066746-ec51-4d7a-8d62-a8a18b2cbadc, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 3b066746-ec51-4d7a-8d62-a8a18b2cbadc; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 3b066746-ec51-4d7a-8d62-a8a18b2cbadc, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:01:27.929Z","duration in milliseconds":22.8363,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:01:27.6485975Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90239,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0489},{"name":"Get PkValue From Stream","duration in milliseconds":0.1612,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":22.4509,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":22.4175,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:00:28.3590776Z","cpu":1.950,"memory":5208932.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0958,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:00:38.3650678Z","cpu":1.484,"memory":5221676.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0825,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:00:48.3711730Z","cpu":2.422,"memory":5221208.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1084,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:00:58.3867232Z","cpu":5.772,"memory":5218140.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1101,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:01:08.3944961Z","cpu":5.538,"memory":5213844.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1319,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:01:18.4049068Z","cpu":13.672,"memory":5227872.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0671,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":22.410800000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":22.407700000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":22.39,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":22.3858,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":21.427500000000002,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:01:27.9454752Z","DurationInMs":15.5614,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"3b066746-ec51-4d7a-8d62-a8a18b2cbadc","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26148762,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26148762,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26148762,"SessionToken":"-1#26148762","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.753","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:01:27 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:01:27.9299138Z", "durationInMs": 0.014},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:01:27.9299278Z", "durationInMs": 18.1943},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:01:27.9481221Z", "durationInMs": 0.1918},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:01:27.9483139Z", "durationInMs": 2.1227},{"event": "Received", "startTimeUtc": "2026-04-14T15:01:27.9504366Z", "durationInMs": 0.1363},{"event": "Completed", "startTimeUtc": "2026-04-14T15:01:27.9505729Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:01:27.9454752Z","lastSend":"2026-04-14T15:01:27.9454752Z","lastReceive":"2026-04-14T15:01:27.9454752Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.29687769999999997,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"3b066746-ec51-4d7a-8d62-a8a18b2cbadc","ResponseTimeUtc":"2026-04-14T15:01:27.9454752Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 3b066746-ec51-4d7a-8d62-a8a18b2cbadc, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26148762","BELatencyInMs":"0.753"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0015}]}
2026-04-14 15:10:29 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 2777518e-c4e5-4325-b26d-f068d552364f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 2777518e-c4e5-4325-b26d-f068d552364f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 2777518e-c4e5-4325-b26d-f068d552364f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 2777518e-c4e5-4325-b26d-f068d552364f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:10:29.124Z","duration in milliseconds":18.7406,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:10:28.4840261Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90339,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0495},{"name":"Get PkValue From Stream","duration in milliseconds":0.21530000000000002,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0023}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":18.3033,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":18.2711,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:09:38.8519910Z","cpu":1.092,"memory":5303160.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1203,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:09:48.8646389Z","cpu":2.262,"memory":5287764.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1058,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:09:58.8780034Z","cpu":0.936,"memory":5285980.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1119,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:10:08.8921416Z","cpu":2.808,"memory":5235536.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1091,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:10:18.9046749Z","cpu":2.184,"memory":5234964.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0843,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:10:28.9059040Z","cpu":3.125,"memory":5235076.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1327,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":2}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":18.2633,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":18.258,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":18.239800000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":18.2378,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":17.255200000000002,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:10:29.1402827Z","DurationInMs":15.625300000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"2777518e-c4e5-4325-b26d-f068d552364f","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149031,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149031,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149031,"SessionToken":"-1#26149031","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.701","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:10:29 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:10:29.1246574Z", "durationInMs": 0.0153},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:10:29.1246727Z", "durationInMs": 14.2498},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:10:29.1389225Z", "durationInMs": 0.1885},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:10:29.1391110Z", "durationInMs": 1.908},{"event": "Received", "startTimeUtc": "2026-04-14T15:10:29.1410190Z", "durationInMs": 0.1365},{"event": "Completed", "startTimeUtc": "2026-04-14T15:10:29.1411555Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:10:29.1246574Z","lastSend":"2026-04-14T15:10:29.1246574Z","lastReceive":"2026-04-14T15:10:29.1246574Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.6562566,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"2777518e-c4e5-4325-b26d-f068d552364f","ResponseTimeUtc":"2026-04-14T15:10:29.1402827Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 2777518e-c4e5-4325-b26d-f068d552364f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149031","BELatencyInMs":"0.701"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 15:11:32 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 40535ec4-e02a-4389-9da5-23f53a6ecf70; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 40535ec4-e02a-4389-9da5-23f53a6ecf70, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 40535ec4-e02a-4389-9da5-23f53a6ecf70; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 40535ec4-e02a-4389-9da5-23f53a6ecf70, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:11:32.302Z","duration in milliseconds":25.535400000000003,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:11:32.1458368Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90346,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0538},{"name":"Get PkValue From Stream","duration in milliseconds":0.1647,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0019}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":25.1325,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":25.096600000000002,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:10:38.9207074Z","cpu":1.248,"memory":5248916.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0336,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:10:48.9306848Z","cpu":1.484,"memory":5252864.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1016,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:10:58.9442679Z","cpu":2.730,"memory":5252392.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1207,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:11:08.9477738Z","cpu":2.813,"memory":5249816.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1035,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:11:18.9594782Z","cpu":1.404,"memory":5209684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.13,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:11:28.9661180Z","cpu":1.716,"memory":5209600.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1076,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":25.089100000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":25.0854,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":25.0594,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":25.055500000000002,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":24.160800000000002,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:11:32.3333390Z","DurationInMs":31.104200000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"40535ec4-e02a-4389-9da5-23f53a6ecf70","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149031,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149031,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149031,"SessionToken":"-1#26149031","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.595","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:11:32 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:11:32.3022348Z", "durationInMs": 0.0168},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:11:32.3022516Z", "durationInMs": 21.2489},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:11:32.3235005Z", "durationInMs": 0.2083},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:11:32.3237088Z", "durationInMs": 1.7706},{"event": "Received", "startTimeUtc": "2026-04-14T15:11:32.3254794Z", "durationInMs": 0.1415},{"event": "Completed", "startTimeUtc": "2026-04-14T15:11:32.3256209Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:11:32.3333390Z","lastSend":"2026-04-14T15:11:32.3333390Z","lastReceive":"2026-04-14T15:11:32.3333390Z"},"requestSizeInBytes":1639,"requestBodySizeInBytes":1187,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.18750219999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"40535ec4-e02a-4389-9da5-23f53a6ecf70","ResponseTimeUtc":"2026-04-14T15:11:32.3333390Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 40535ec4-e02a-4389-9da5-23f53a6ecf70, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149031","BELatencyInMs":"0.595"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 15:28:19 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c43428ad-7469-44f1-8f41-b81868236d49; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c43428ad-7469-44f1-8f41-b81868236d49, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c43428ad-7469-44f1-8f41-b81868236d49; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c43428ad-7469-44f1-8f41-b81868236d49, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:28:19.868Z","duration in milliseconds":26.4323,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:28:19.6805203Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90538,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0456},{"name":"Get PkValue From Stream","duration in milliseconds":0.1802,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":26.021700000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":25.979400000000002,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:27:29.7682585Z","cpu":1.563,"memory":5183624.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1066,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:27:39.7763378Z","cpu":2.344,"memory":5183832.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1099,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:27:49.7831072Z","cpu":3.822,"memory":5193224.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1089,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:27:59.7983904Z","cpu":5.148,"memory":5251272.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.099,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:28:09.8048391Z","cpu":2.422,"memory":5245092.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.101,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:28:19.8056937Z","cpu":2.891,"memory":5196124.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0999,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":2}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":25.9709,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":25.966700000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":25.947300000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":25.9449,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":24.939500000000002,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:28:19.8837140Z","DurationInMs":15.6905,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c43428ad-7469-44f1-8f41-b81868236d49","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149080,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149080,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149080,"SessionToken":"-1#26149080","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.925","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:28:19 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:28:19.8680235Z", "durationInMs": 0.0153},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:28:19.8680388Z", "durationInMs": 21.0744},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:28:19.8891132Z", "durationInMs": 0.213},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:28:19.8893262Z", "durationInMs": 2.6706},{"event": "Received", "startTimeUtc": "2026-04-14T15:28:19.8919968Z", "durationInMs": 0.153},{"event": "Completed", "startTimeUtc": "2026-04-14T15:28:19.8921498Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:28:19.8837140Z","lastSend":"2026-04-14T15:28:19.8837140Z","lastReceive":"2026-04-14T15:28:19.8837140Z"},"requestSizeInBytes":1611,"requestBodySizeInBytes":1155,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.20319369999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c43428ad-7469-44f1-8f41-b81868236d49","ResponseTimeUtc":"2026-04-14T15:28:19.8837140Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c43428ad-7469-44f1-8f41-b81868236d49, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149080","BELatencyInMs":"0.925"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 15:28:36 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 06384b74-21d8-44e4-96d6-1d2497dcb65b; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 06384b74-21d8-44e4-96d6-1d2497dcb65b, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 06384b74-21d8-44e4-96d6-1d2497dcb65b; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 06384b74-21d8-44e4-96d6-1d2497dcb65b, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:28:36.445Z","duration in milliseconds":4.0681,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:28:35.6641099Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90541,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0524},{"name":"Get PkValue From Stream","duration in milliseconds":0.1699,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0042000000000000006}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":3.6781,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":3.6510000000000002,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:27:39.7763378Z","cpu":2.344,"memory":5183832.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1099,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:27:49.7831072Z","cpu":3.822,"memory":5193224.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1089,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:27:59.7983904Z","cpu":5.148,"memory":5251272.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.099,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:28:09.8048391Z","cpu":2.422,"memory":5245092.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.101,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:28:19.8056937Z","cpu":2.891,"memory":5196124.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0999,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":2},{"dateUtc":"2026-04-14T15:28:29.8187460Z","cpu":0.936,"memory":5207544.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0511,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":3.6449000000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":3.6399000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":3.6244,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":3.6226000000000003,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":2.7417000000000002,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:28:36.4453342Z","DurationInMs":0,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"06384b74-21d8-44e4-96d6-1d2497dcb65b","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149080,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149080,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149080,"SessionToken":"-1#26149080","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.596","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:28:36 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:28:36.4453342Z", "durationInMs": 0.0041},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:28:36.4453383Z", "durationInMs": 0.0035},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:28:36.4453418Z", "durationInMs": 0.184},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:28:36.4455258Z", "durationInMs": 1.7439},{"event": "Received", "startTimeUtc": "2026-04-14T15:28:36.4472697Z", "durationInMs": 0.0665},{"event": "Completed", "startTimeUtc": "2026-04-14T15:28:36.4473362Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:28:36.4296523Z","lastSend":"2026-04-14T15:28:36.4296523Z","lastReceive":"2026-04-14T15:28:36.4453342Z"},"requestSizeInBytes":1606,"requestBodySizeInBytes":1154,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.7812243,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"06384b74-21d8-44e4-96d6-1d2497dcb65b","ResponseTimeUtc":"2026-04-14T15:28:36.4453342Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 06384b74-21d8-44e4-96d6-1d2497dcb65b, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149080","BELatencyInMs":"0.596"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 15:36:07 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d75ba625-b680-46a9-9048-598ed988f7b2; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d75ba625-b680-46a9-9048-598ed988f7b2, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d75ba625-b680-46a9-9048-598ed988f7b2; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d75ba625-b680-46a9-9048-598ed988f7b2, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:36:07.273Z","duration in milliseconds":43.876200000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:36:06.5077914Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90586,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0471},{"name":"Get PkValue From Stream","duration in milliseconds":7.4107,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0036000000000000003},{"name":"Read Collection","duration in milliseconds":7.2505000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T15:36:07.2735030Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"f64bff7b-f236-442c-9840-b541804a3f25","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.2449,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.2125,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:35:10.1844969Z","cpu":3.354,"memory":5285596.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1198,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:35:20.2003158Z","cpu":1.638,"memory":5284020.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1183,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:35:30.2052556Z","cpu":1.094,"memory":5282352.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0974,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:35:40.2136186Z","cpu":3.198,"memory":5281940.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0992,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:35:50.2265315Z","cpu":1.797,"memory":5278236.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1466,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:36:00.2424118Z","cpu":1.794,"memory":5241640.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0912,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.205,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.1999,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.1774,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.175200000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.2451,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T15:36:07.2890585Z","EndTimeUTC":"2026-04-14T15:36:07.2890585Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:36:07.3203059Z","DurationInMs":31.247400000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"d75ba625-b680-46a9-9048-598ed988f7b2","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149289,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149289,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149289,"SessionToken":"-1#26149289","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.666","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:36:07 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:36:07.2890585Z", "durationInMs": 0.0086},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:36:07.2890671Z", "durationInMs": 18.5412},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:36:07.3076083Z", "durationInMs": 0.2728},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:36:07.3078811Z", "durationInMs": 1.8831},{"event": "Received", "startTimeUtc": "2026-04-14T15:36:07.3097642Z", "durationInMs": 0.1399},{"event": "Completed", "startTimeUtc": "2026-04-14T15:36:07.3099041Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:36:07.3047351Z","lastSend":"2026-04-14T15:36:07.3047351Z","lastReceive":"2026-04-14T15:36:07.3047351Z"},"requestSizeInBytes":1620,"requestBodySizeInBytes":1172,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T15:36:07.2735030Z","DurationInMs":15.5555,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"f0986d52-7996-4d59-b401-70574f5bacb2","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T15:36:07.2890585Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"8f182de7-2c38-4424-a9ac-bf9da90f3f2a","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T15:36:07.2890585Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"d75ba625-b680-46a9-9048-598ed988f7b2","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.81251449999999992,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"d75ba625-b680-46a9-9048-598ed988f7b2","ResponseTimeUtc":"2026-04-14T15:36:07.3203059Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: d75ba625-b680-46a9-9048-598ed988f7b2, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149289","BELatencyInMs":"0.666"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 15:37:59 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: f0f4737c-1428-44c0-87d1-76e5c02c409a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: f0f4737c-1428-44c0-87d1-76e5c02c409a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: f0f4737c-1428-44c0-87d1-76e5c02c409a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: f0f4737c-1428-44c0-87d1-76e5c02c409a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:37:59.133Z","duration in milliseconds":4.6586,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:37:58.8362661Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90604,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.050300000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":0.1578,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0033}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":4.2866,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":4.2603,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:37:00.2915015Z","cpu":2.418,"memory":5249432.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2198,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:37:10.3031222Z","cpu":2.422,"memory":5256172.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2269,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:37:20.3102019Z","cpu":1.248,"memory":5253940.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0918,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:37:30.3260873Z","cpu":4.992,"memory":5231620.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0651,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:37:40.3284964Z","cpu":2.266,"memory":5264848.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1097,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:37:50.3332132Z","cpu":0.938,"memory":5275492.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1233,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":4.2542,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":4.2499,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":4.2341000000000006,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":4.232,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":3.3040000000000003,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:37:59.1331461Z","DurationInMs":0,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"f0f4737c-1428-44c0-87d1-76e5c02c409a","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149289,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149289,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149289,"SessionToken":"-1#26149289","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.793","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:37:59 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:37:59.1331461Z", "durationInMs": 0.0028},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:37:59.1331489Z", "durationInMs": 0.0049},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:37:59.1331538Z", "durationInMs": 0.2259},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:37:59.1333797Z", "durationInMs": 2.3241},{"event": "Received", "startTimeUtc": "2026-04-14T15:37:59.1357038Z", "durationInMs": 0.0667},{"event": "Completed", "startTimeUtc": "2026-04-14T15:37:59.1357705Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:37:59.1331461Z","lastSend":"2026-04-14T15:37:59.1331461Z","lastReceive":"2026-04-14T15:37:59.1331461Z"},"requestSizeInBytes":1604,"requestBodySizeInBytes":1156,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.29688,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"f0f4737c-1428-44c0-87d1-76e5c02c409a","ResponseTimeUtc":"2026-04-14T15:37:59.1331461Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: f0f4737c-1428-44c0-87d1-76e5c02c409a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149289","BELatencyInMs":"0.793"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 15:41:42 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 0797184f-4ebb-4756-82ed-91e9452df7ee; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 0797184f-4ebb-4756-82ed-91e9452df7ee, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 0797184f-4ebb-4756-82ed-91e9452df7ee; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 0797184f-4ebb-4756-82ed-91e9452df7ee, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:41:42.540Z","duration in milliseconds":21.3185,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:41:41.8842054Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90696,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0509},{"name":"Get PkValue From Stream","duration in milliseconds":0.2122,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":20.8229,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":20.7843,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:40:50.5082067Z","cpu":1.953,"memory":5222580.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0898,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:41:00.5152913Z","cpu":2.496,"memory":5236240.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0972,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:41:10.5260320Z","cpu":6.641,"memory":5234700.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1186,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:41:20.5394254Z","cpu":2.808,"memory":5234152.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2192,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:41:30.5465904Z","cpu":3.354,"memory":5234752.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1101,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:41:40.5556991Z","cpu":1.875,"memory":5231336.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0921,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":20.7732,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":20.7682,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":20.7441,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":20.741600000000002,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":19.2942,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:41:42.5560947Z","DurationInMs":15.6239,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"0797184f-4ebb-4756-82ed-91e9452df7ee","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149304,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149304,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149304,"SessionToken":"-1#26149304","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.933","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:41:42 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:41:42.5404708Z", "durationInMs": 0.0153},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:41:42.5404861Z", "durationInMs": 15.88},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:41:42.5563661Z", "durationInMs": 0.2289},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:41:42.5565950Z", "durationInMs": 2.2324},{"event": "Received", "startTimeUtc": "2026-04-14T15:41:42.5588274Z", "durationInMs": 0.1449},{"event": "Completed", "startTimeUtc": "2026-04-14T15:41:42.5589723Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:41:42.5404708Z","lastSend":"2026-04-14T15:41:42.5404708Z","lastReceive":"2026-04-14T15:41:42.5404708Z"},"requestSizeInBytes":1653,"requestBodySizeInBytes":1203,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.6718893,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"0797184f-4ebb-4756-82ed-91e9452df7ee","ResponseTimeUtc":"2026-04-14T15:41:42.5560947Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 0797184f-4ebb-4756-82ed-91e9452df7ee, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149304","BELatencyInMs":"0.933"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 15:46:07 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 9a9fac1f-2227-4296-a6dd-574816467971; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 9a9fac1f-2227-4296-a6dd-574816467971, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 9a9fac1f-2227-4296-a6dd-574816467971; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 9a9fac1f-2227-4296-a6dd-574816467971, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:46:07.479Z","duration in milliseconds":44.788000000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:46:06.0103251Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90739,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0473},{"name":"Get PkValue From Stream","duration in milliseconds":6.3638,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.1831000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T15:46:07.4791139Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"43933c9a-27a3-4304-8ccb-9616fa7d134c","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":38.199600000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":38.1616,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:45:10.7018980Z","cpu":9.516,"memory":5213104.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0538,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:45:20.7048544Z","cpu":4.766,"memory":5194064.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1212,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:45:30.7159013Z","cpu":0.468,"memory":5192620.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2673,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:45:40.7184542Z","cpu":2.266,"memory":5192120.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0935,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:45:50.7263688Z","cpu":2.418,"memory":5216592.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0998,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:46:00.7390767Z","cpu":4.056,"memory":5183932.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0965,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":38.152,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":38.1488,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":38.1296,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":38.127,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":37.1436,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T15:46:07.4947305Z","EndTimeUTC":"2026-04-14T15:46:07.4947305Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:46:07.5259803Z","DurationInMs":31.2498,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"9a9fac1f-2227-4296-a6dd-574816467971","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149304,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149304,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149304,"SessionToken":"-1#26149304","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.809","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:46:07 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:46:07.4947305Z", "durationInMs": 0.0083},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:46:07.4947388Z", "durationInMs": 18.7754},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:46:07.5135142Z", "durationInMs": 0.2057},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:46:07.5137199Z", "durationInMs": 2.1136},{"event": "Received", "startTimeUtc": "2026-04-14T15:46:07.5158335Z", "durationInMs": 0.1487},{"event": "Completed", "startTimeUtc": "2026-04-14T15:46:07.5159822Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:46:07.5259803Z","lastSend":"2026-04-14T15:46:07.5259803Z","lastReceive":"2026-04-14T15:46:07.5259803Z"},"requestSizeInBytes":1596,"requestBodySizeInBytes":1148,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T15:46:07.4947305Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"f38fb679-c025-431e-a44a-434ae67892a5","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T15:46:07.4947305Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"886bcdc0-1a5a-4e03-b5f3-999d6e840ab4","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T15:46:07.4947305Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"9a9fac1f-2227-4296-a6dd-574816467971","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":1.5156551999999999,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"9a9fac1f-2227-4296-a6dd-574816467971","ResponseTimeUtc":"2026-04-14T15:46:07.5259803Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 9a9fac1f-2227-4296-a6dd-574816467971, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149304","BELatencyInMs":"0.809"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001}]}
2026-04-14 15:46:20 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 0682a856-2c2c-4920-847b-a72b17ed590c; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 0682a856-2c2c-4920-847b-a72b17ed590c, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 0682a856-2c2c-4920-847b-a72b17ed590c; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 0682a856-2c2c-4920-847b-a72b17ed590c, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:46:20.205Z","duration in milliseconds":18.686600000000002,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:46:19.9090655Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90749,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.053000000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":0.15860000000000002,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0015}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":18.3064,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":18.271800000000002,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:45:20.7048544Z","cpu":4.766,"memory":5194064.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1212,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:45:30.7159013Z","cpu":0.468,"memory":5192620.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2673,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:45:40.7184542Z","cpu":2.266,"memory":5192120.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0935,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:45:50.7263688Z","cpu":2.418,"memory":5216592.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0998,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:46:00.7390767Z","cpu":4.056,"memory":5183932.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0965,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:46:10.7436203Z","cpu":8.438,"memory":5183944.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0871,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":18.265700000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":18.2576,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":18.2396,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":18.2372,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":17.2843,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:46:20.2215642Z","DurationInMs":15.613700000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"0682a856-2c2c-4920-847b-a72b17ed590c","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149304,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149304,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149304,"SessionToken":"-1#26149304","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.548","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:46:20 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:46:20.2059505Z", "durationInMs": 0.0124},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:46:20.2059629Z", "durationInMs": 14.3019},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:46:20.2202648Z", "durationInMs": 0.1897},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:46:20.2204545Z", "durationInMs": 1.8825},{"event": "Received", "startTimeUtc": "2026-04-14T15:46:20.2223370Z", "durationInMs": 0.1308},{"event": "Completed", "startTimeUtc": "2026-04-14T15:46:20.2224678Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:46:20.2215642Z","lastSend":"2026-04-14T15:46:20.2215642Z","lastReceive":"2026-04-14T15:46:20.2215642Z"},"requestSizeInBytes":1606,"requestBodySizeInBytes":1156,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.31249869999999996,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"0682a856-2c2c-4920-847b-a72b17ed590c","ResponseTimeUtc":"2026-04-14T15:46:20.2215642Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 0682a856-2c2c-4920-847b-a72b17ed590c, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149304","BELatencyInMs":"0.548"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 15:47:04 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 8b8aab29-c4d3-468e-a23d-dead6ae9f3ae; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 8b8aab29-c4d3-468e-a23d-dead6ae9f3ae, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 8b8aab29-c4d3-468e-a23d-dead6ae9f3ae; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 8b8aab29-c4d3-468e-a23d-dead6ae9f3ae, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:47:04.672Z","duration in milliseconds":24.0791,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:47:04.2821023Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90767,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.051000000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":0.16140000000000002,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":23.6883,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":23.650100000000002,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:46:10.7436203Z","cpu":8.438,"memory":5183944.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0871,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:46:20.7529116Z","cpu":2.891,"memory":5188304.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1041,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:46:30.7668707Z","cpu":3.432,"memory":5185832.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0862,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:46:40.7707288Z","cpu":2.496,"memory":5186976.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0913,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:46:50.7858461Z","cpu":1.563,"memory":5186760.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1094,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:47:00.7958304Z","cpu":3.042,"memory":5199632.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.128,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":23.6416,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":23.6373,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":23.6187,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":23.6161,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":22.6031,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:47:04.7038699Z","DurationInMs":31.2616,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"8b8aab29-c4d3-468e-a23d-dead6ae9f3ae","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149304,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149304,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149304,"SessionToken":"-1#26149304","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.614","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:47:04 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:47:04.6726083Z", "durationInMs": 0.0158},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:47:04.6726241Z", "durationInMs": 19.3583},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:47:04.6919824Z", "durationInMs": 0.2427},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:47:04.6922251Z", "durationInMs": 2.0236},{"event": "Received", "startTimeUtc": "2026-04-14T15:47:04.6942487Z", "durationInMs": 0.1549},{"event": "Completed", "startTimeUtc": "2026-04-14T15:47:04.6944036Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:47:04.6882337Z","lastSend":"2026-04-14T15:47:04.6882337Z","lastReceive":"2026-04-14T15:47:04.7038699Z"},"requestSizeInBytes":1605,"requestBodySizeInBytes":1153,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.42176759999999996,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"8b8aab29-c4d3-468e-a23d-dead6ae9f3ae","ResponseTimeUtc":"2026-04-14T15:47:04.7038699Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 8b8aab29-c4d3-468e-a23d-dead6ae9f3ae, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149304","BELatencyInMs":"0.614"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 15:50:59 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c7c602e4-0526-4634-81e9-f5e2bcff08e9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c7c602e4-0526-4634-81e9-f5e2bcff08e9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c7c602e4-0526-4634-81e9-f5e2bcff08e9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c7c602e4-0526-4634-81e9-f5e2bcff08e9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:50:59.416Z","duration in milliseconds":57.6783,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:50:58.5723094Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90782,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0477},{"name":"Get PkValue From Stream","duration in milliseconds":8.7136,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0015},{"name":"Read Collection","duration in milliseconds":8.549100000000001,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T15:50:59.4161318Z","DurationInMs":15.6448,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"8f7e448e-0b88-450c-b6ea-211ebf453e05","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":48.7215,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":48.6882,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:50:00.9442166Z","cpu":3.516,"memory":5343636.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0998,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:50:10.9498343Z","cpu":3.198,"memory":5292596.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0762,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:50:20.9544463Z","cpu":2.266,"memory":5290868.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1213,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:50:30.9627242Z","cpu":1.170,"memory":5290916.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1245,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:50:40.9775232Z","cpu":0.859,"memory":5290136.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1198,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:50:50.9823216Z","cpu":1.638,"memory":5289668.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1386,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":48.678900000000006,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":48.674800000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":48.6527,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":48.6499,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":47.680600000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T15:50:59.4317766Z","EndTimeUTC":"2026-04-14T15:50:59.4473282Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:50:59.4785830Z","DurationInMs":31.254800000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c7c602e4-0526-4634-81e9-f5e2bcff08e9","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149316,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149316,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149316,"SessionToken":"-1#26149316","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.647","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:50:59 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:50:59.4473282Z", "durationInMs": 0.0078},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:50:59.4473360Z", "durationInMs": 29.7919},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:50:59.4771279Z", "durationInMs": 0.2085},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:50:59.4773364Z", "durationInMs": 2.1052},{"event": "Received", "startTimeUtc": "2026-04-14T15:50:59.4794416Z", "durationInMs": 0.1419},{"event": "Completed", "startTimeUtc": "2026-04-14T15:50:59.4795835Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:50:59.4629839Z","lastSend":"2026-04-14T15:50:59.4629839Z","lastReceive":"2026-04-14T15:50:59.4629839Z"},"requestSizeInBytes":1622,"requestBodySizeInBytes":1170,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T15:50:59.4317766Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"bd53cf95-4528-44cb-b255-2ff142dae5ff","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T15:50:59.4317766Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"779dbb75-1a4a-46e8-86fe-5a831828cb08","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T15:50:59.4317766Z","DurationInMs":15.5516,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"c7c602e4-0526-4634-81e9-f5e2bcff08e9","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.9062736,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c7c602e4-0526-4634-81e9-f5e2bcff08e9","ResponseTimeUtc":"2026-04-14T15:50:59.4785830Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c7c602e4-0526-4634-81e9-f5e2bcff08e9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149316","BELatencyInMs":"0.647"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 15:52:57 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: faab42b6-b12b-416b-9741-3c5a0a2d9031; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: faab42b6-b12b-416b-9741-3c5a0a2d9031, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: faab42b6-b12b-416b-9741-3c5a0a2d9031; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: faab42b6-b12b-416b-9741-3c5a0a2d9031, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:52:57.220Z","duration in milliseconds":45.497800000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:52:56.6579854Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90793,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.044500000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":8.1982,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":8.0517,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T15:52:57.2204975Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"31c07d1b-85c9-4c2c-990c-462b6ba72358","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.084900000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.050000000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:52:01.0438945Z","cpu":2.340,"memory":5252660.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1191,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:52:11.0482139Z","cpu":5.234,"memory":5239688.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0936,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:52:21.0501962Z","cpu":0.625,"memory":5264232.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0945,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:52:31.0508958Z","cpu":1.406,"memory":5254700.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1111,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:52:41.0648953Z","cpu":2.184,"memory":5252384.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2024,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:52:51.0755861Z","cpu":2.496,"memory":5250820.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2528,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.0428,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.0388,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.02,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.0176,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.0499,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T15:52:57.2361235Z","EndTimeUTC":"2026-04-14T15:52:57.2361235Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:52:57.2673728Z","DurationInMs":31.2493,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"faab42b6-b12b-416b-9741-3c5a0a2d9031","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149358,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149358,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149358,"SessionToken":"-1#26149358","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.772","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:52:57 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:52:57.2361235Z", "durationInMs": 0.0084},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:52:57.2361319Z", "durationInMs": 19.0009},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:52:57.2551328Z", "durationInMs": 0.2114},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:52:57.2553442Z", "durationInMs": 2.0408},{"event": "Received", "startTimeUtc": "2026-04-14T15:52:57.2573850Z", "durationInMs": 0.1438},{"event": "Completed", "startTimeUtc": "2026-04-14T15:52:57.2575288Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:52:57.2517543Z","lastSend":"2026-04-14T15:52:57.2517543Z","lastReceive":"2026-04-14T15:52:57.2517543Z"},"requestSizeInBytes":1596,"requestBodySizeInBytes":1148,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T15:52:57.2204975Z","DurationInMs":15.626000000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"99d387dc-cc4c-48d3-bd5d-ba91c25aeeb5","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T15:52:57.2361235Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"40786c1b-1539-47b8-8401-11459e6240bd","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T15:52:57.2361235Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"faab42b6-b12b-416b-9741-3c5a0a2d9031","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.6093874,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"faab42b6-b12b-416b-9741-3c5a0a2d9031","ResponseTimeUtc":"2026-04-14T15:52:57.2673728Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: faab42b6-b12b-416b-9741-3c5a0a2d9031, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149358","BELatencyInMs":"0.772"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 15:56:33 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 397059f2-647b-4a22-ad61-a17d72fa69fa; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 397059f2-647b-4a22-ad61-a17d72fa69fa, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 397059f2-647b-4a22-ad61-a17d72fa69fa; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 397059f2-647b-4a22-ad61-a17d72fa69fa, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:56:33.083Z","duration in milliseconds":22.0743,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:56:32.6612723Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90841,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0541},{"name":"Get PkValue From Stream","duration in milliseconds":0.16,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0032}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":21.6946,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":21.665000000000003,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:55:41.2281373Z","cpu":0.703,"memory":5217032.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1221,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:55:51.2362695Z","cpu":0.702,"memory":5214876.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0962,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:56:01.2402048Z","cpu":6.953,"memory":5175228.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1089,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:56:11.2525363Z","cpu":8.424,"memory":5169612.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0745,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:56:21.2627766Z","cpu":0.938,"memory":5173448.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.107,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:56:31.2746517Z","cpu":1.170,"memory":5171316.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1005,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":21.657400000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":21.6536,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":21.6374,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":21.6355,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":20.678700000000003,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:56:33.0988383Z","DurationInMs":15.6371,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"397059f2-647b-4a22-ad61-a17d72fa69fa","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149358,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149358,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149358,"SessionToken":"-1#26149358","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.731","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:56:33 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:56:33.0832012Z", "durationInMs": 0.015},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:56:33.0832162Z", "durationInMs": 17.4979},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:56:33.1007141Z", "durationInMs": 0.1856},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:56:33.1008997Z", "durationInMs": 2.0558},{"event": "Received", "startTimeUtc": "2026-04-14T15:56:33.1029555Z", "durationInMs": 0.1389},{"event": "Completed", "startTimeUtc": "2026-04-14T15:56:33.1030944Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:56:33.0988383Z","lastSend":"2026-04-14T15:56:33.0988383Z","lastReceive":"2026-04-14T15:56:33.0988383Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1188,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.43756599999999996,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"397059f2-647b-4a22-ad61-a17d72fa69fa","ResponseTimeUtc":"2026-04-14T15:56:33.0988383Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 397059f2-647b-4a22-ad61-a17d72fa69fa, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149358","BELatencyInMs":"0.731"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 15:57:04 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 71bc0cad-4408-4756-a27b-b88b6ca7492d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 71bc0cad-4408-4756-a27b-b88b6ca7492d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 71bc0cad-4408-4756-a27b-b88b6ca7492d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 71bc0cad-4408-4756-a27b-b88b6ca7492d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:57:04.193Z","duration in milliseconds":20.0573,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:57:03.9122140Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90863,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0536},{"name":"Get PkValue From Stream","duration in milliseconds":0.168,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0015}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":19.6609,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":19.6283,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:56:11.2525363Z","cpu":8.424,"memory":5169612.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0745,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:56:21.2627766Z","cpu":0.938,"memory":5173448.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.107,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:56:31.2746517Z","cpu":1.170,"memory":5171316.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1005,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:56:41.2848159Z","cpu":2.886,"memory":5170624.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":1.5968,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:56:51.2892499Z","cpu":3.594,"memory":5171264.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.08,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:57:01.2934848Z","cpu":3.516,"memory":5183800.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1133,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":19.6216,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":19.617800000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":19.601300000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":19.5978,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":18.614800000000002,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:57:04.2247149Z","DurationInMs":31.250600000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"71bc0cad-4408-4756-a27b-b88b6ca7492d","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149358,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149358,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149358,"SessionToken":"-1#26149358","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.705","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:57:04 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:57:04.1934643Z", "durationInMs": 0.015},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:57:04.1934793Z", "durationInMs": 15.3562},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:57:04.2088355Z", "durationInMs": 0.1965},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:57:04.2090320Z", "durationInMs": 2.1159},{"event": "Received", "startTimeUtc": "2026-04-14T15:57:04.2111479Z", "durationInMs": 0.1409},{"event": "Completed", "startTimeUtc": "2026-04-14T15:57:04.2112888Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:57:04.2090992Z","lastSend":"2026-04-14T15:57:04.2090992Z","lastReceive":"2026-04-14T15:57:04.2090992Z"},"requestSizeInBytes":1668,"requestBodySizeInBytes":1214,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.31250089999999997,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"71bc0cad-4408-4756-a27b-b88b6ca7492d","ResponseTimeUtc":"2026-04-14T15:57:04.2247149Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 71bc0cad-4408-4756-a27b-b88b6ca7492d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149358","BELatencyInMs":"0.705"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0014}]}
2026-04-14 15:58:11 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 1c27c36f-d4f5-4f50-a6a8-cd5d52079fb4; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 1c27c36f-d4f5-4f50-a6a8-cd5d52079fb4, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 1c27c36f-d4f5-4f50-a6a8-cd5d52079fb4; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 1c27c36f-d4f5-4f50-a6a8-cd5d52079fb4, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:58:11.434Z","duration in milliseconds":43.6393,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:58:11.3098357Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90865,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0413},{"name":"Get PkValue From Stream","duration in milliseconds":5.9383,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":5.787,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T15:58:11.4348929Z","DurationInMs":15.628,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"af6d368e-7312-43de-8634-2e6958c52f72","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.490500000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.4593,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:57:21.3126302Z","cpu":0.781,"memory":5187172.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0877,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:57:31.3211591Z","cpu":0.468,"memory":5187252.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0875,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:57:41.3227258Z","cpu":2.813,"memory":5189248.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1015,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:57:51.3523257Z","cpu":3.505,"memory":5206164.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1136,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:58:01.3607079Z","cpu":1.484,"memory":5284024.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0892,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:58:11.3724116Z","cpu":1.014,"memory":5274472.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1056,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.450900000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.446600000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.426100000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.4238,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.581700000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T15:58:11.4505209Z","EndTimeUTC":"2026-04-14T15:58:11.4505209Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:58:11.4817182Z","DurationInMs":31.197300000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"1c27c36f-d4f5-4f50-a6a8-cd5d52079fb4","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149358,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149358,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149358,"SessionToken":"-1#26149358","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.62","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:58:11 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:58:11.4505209Z", "durationInMs": 0.0077},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:58:11.4505286Z", "durationInMs": 20.6308},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:58:11.4711594Z", "durationInMs": 0.2325},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:58:11.4713919Z", "durationInMs": 1.9495},{"event": "Received", "startTimeUtc": "2026-04-14T15:58:11.4733414Z", "durationInMs": 0.1441},{"event": "Completed", "startTimeUtc": "2026-04-14T15:58:11.4734855Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:58:11.4817182Z","lastSend":"2026-04-14T15:58:11.4817182Z","lastReceive":"2026-04-14T15:58:11.4817182Z"},"requestSizeInBytes":1638,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T15:58:11.4505209Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"48d4d64c-7d12-4ae3-9646-c1615296a9dd","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T15:58:11.4505209Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"991767ec-7b43-4854-8558-f7b9f11f9c1d","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T15:58:11.4505209Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"1c27c36f-d4f5-4f50-a6a8-cd5d52079fb4","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.18750619999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"1c27c36f-d4f5-4f50-a6a8-cd5d52079fb4","ResponseTimeUtc":"2026-04-14T15:58:11.4817182Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 1c27c36f-d4f5-4f50-a6a8-cd5d52079fb4, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149358","BELatencyInMs":"0.62"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0015}]}
2026-04-14 15:59:37 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 9055a7c4-be69-44d1-86da-fe5202abdecb; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 9055a7c4-be69-44d1-86da-fe5202abdecb, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 9055a7c4-be69-44d1-86da-fe5202abdecb; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 9055a7c4-be69-44d1-86da-fe5202abdecb, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:59:37.126Z","duration in milliseconds":51.924,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:59:36.9859700Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90894,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.049},{"name":"Get PkValue From Stream","duration in milliseconds":12.447700000000001,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.006},{"name":"Read Collection","duration in milliseconds":12.2782,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T15:59:37.1266048Z","DurationInMs":15.7025,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"0eec89dc-e20f-4f4b-965b-b59b2830ec9d","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":39.24,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":39.204,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:58:41.3913361Z","cpu":1.563,"memory":5273268.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0951,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:58:51.4046081Z","cpu":2.418,"memory":5270232.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1242,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:01.4135532Z","cpu":3.588,"memory":5268720.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1086,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:11.4215566Z","cpu":1.953,"memory":5264392.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1424,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:21.4332683Z","cpu":2.262,"memory":5261876.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.094,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:31.4462762Z","cpu":1.248,"memory":5255532.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1012,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":39.1944,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":39.1895,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":39.1672,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":39.1649,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":38.1749,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T15:59:37.1423073Z","EndTimeUTC":"2026-04-14T15:59:37.1578997Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:59:37.1735607Z","DurationInMs":15.661000000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"9055a7c4-be69-44d1-86da-fe5202abdecb","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149358,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149358,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149358,"SessionToken":"-1#26149358","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.676","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:59:37 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:59:37.1578997Z", "durationInMs": 0.0121},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:59:37.1579118Z", "durationInMs": 19.9898},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:59:37.1779016Z", "durationInMs": 0.3122},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:59:37.1782138Z", "durationInMs": 2.3193},{"event": "Received", "startTimeUtc": "2026-04-14T15:59:37.1805331Z", "durationInMs": 0.1351},{"event": "Completed", "startTimeUtc": "2026-04-14T15:59:37.1806682Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:59:37.1735607Z","lastSend":"2026-04-14T15:59:37.1735607Z","lastReceive":"2026-04-14T15:59:37.1735607Z"},"requestSizeInBytes":1630,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T15:59:37.1423073Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"ccd21232-0fc3-49a8-8bda-fe50b28da4b9","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T15:59:37.1423073Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"25f52bb5-22da-4b9f-b8f2-f562e6c3318c","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T15:59:37.1423073Z","DurationInMs":15.592400000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"9055a7c4-be69-44d1-86da-fe5202abdecb","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1875907,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"9055a7c4-be69-44d1-86da-fe5202abdecb","ResponseTimeUtc":"2026-04-14T15:59:37.1735607Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 9055a7c4-be69-44d1-86da-fe5202abdecb, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149358","BELatencyInMs":"0.676"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 15:59:51 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 9e48e68c-9d5b-470b-b38a-7da88e61d45e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 9e48e68c-9d5b-470b-b38a-7da88e61d45e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 9e48e68c-9d5b-470b-b38a-7da88e61d45e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 9e48e68c-9d5b-470b-b38a-7da88e61d45e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:59:51.557Z","duration in milliseconds":26.1229,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:59:51.1982818Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90901,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0597},{"name":"Get PkValue From Stream","duration in milliseconds":0.1691,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":25.710700000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":25.666800000000002,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:59:01.4135532Z","cpu":3.588,"memory":5268720.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1086,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:11.4215566Z","cpu":1.953,"memory":5264392.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1424,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:21.4332683Z","cpu":2.262,"memory":5261876.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.094,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:31.4462762Z","cpu":1.248,"memory":5255532.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1012,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:41.4600527Z","cpu":2.652,"memory":5255096.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1164,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:51.4637978Z","cpu":1.641,"memory":5256664.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0972,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":25.657700000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":25.650100000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":25.6313,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":25.628500000000003,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":24.617800000000003,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:59:51.5731753Z","DurationInMs":15.627,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"9e48e68c-9d5b-470b-b38a-7da88e61d45e","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149358,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149358,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149358,"SessionToken":"-1#26149358","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.495","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:59:51 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:59:51.5575483Z", "durationInMs": 0.0149},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:59:51.5575632Z", "durationInMs": 21.2076},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:59:51.5787708Z", "durationInMs": 0.2101},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:59:51.5789809Z", "durationInMs": 2.2329},{"event": "Received", "startTimeUtc": "2026-04-14T15:59:51.5812138Z", "durationInMs": 0.1481},{"event": "Completed", "startTimeUtc": "2026-04-14T15:59:51.5813619Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:59:51.5731753Z","lastSend":"2026-04-14T15:59:51.5731753Z","lastReceive":"2026-04-14T15:59:51.5731753Z"},"requestSizeInBytes":1604,"requestBodySizeInBytes":1154,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.3748935,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"9e48e68c-9d5b-470b-b38a-7da88e61d45e","ResponseTimeUtc":"2026-04-14T15:59:51.5731753Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 9e48e68c-9d5b-470b-b38a-7da88e61d45e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149358","BELatencyInMs":"0.495"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 15:59:57 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 0e247651-fc60-4af8-99a3-14444cb0b045; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 0e247651-fc60-4af8-99a3-14444cb0b045, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 0e247651-fc60-4af8-99a3-14444cb0b045; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 0e247651-fc60-4af8-99a3-14444cb0b045, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T15:59:57.822Z","duration in milliseconds":44.191700000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T15:59:57.6977751Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90902,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.048},{"name":"Get PkValue From Stream","duration in milliseconds":6.9125000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0037},{"name":"Read Collection","duration in milliseconds":6.7558000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T15:59:57.8228604Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"c222219c-db94-461c-99f4-a520c95d4f30","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.0508,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.02,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:59:01.4135532Z","cpu":3.588,"memory":5268720.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1086,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:11.4215566Z","cpu":1.953,"memory":5264392.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1424,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:21.4332683Z","cpu":2.262,"memory":5261876.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.094,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:31.4462762Z","cpu":1.248,"memory":5255532.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1012,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:41.4600527Z","cpu":2.652,"memory":5255096.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1164,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:51.4637978Z","cpu":1.641,"memory":5256664.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0972,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.012,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.008500000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.9885,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.9861,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.119,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T15:59:57.8384058Z","EndTimeUTC":"2026-04-14T15:59:57.8384058Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T15:59:57.8696512Z","DurationInMs":31.2454,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"0e247651-fc60-4af8-99a3-14444cb0b045","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149358,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149358,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149358,"SessionToken":"-1#26149358","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.711","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 3:59:57 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T15:59:57.8384058Z", "durationInMs": 0.0079},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T15:59:57.8384137Z", "durationInMs": 17.6671},{"event": "Pipelined", "startTimeUtc": "2026-04-14T15:59:57.8560808Z", "durationInMs": 0.2072},{"event": "Transit Time", "startTimeUtc": "2026-04-14T15:59:57.8562880Z", "durationInMs": 2.1797},{"event": "Received", "startTimeUtc": "2026-04-14T15:59:57.8584677Z", "durationInMs": 0.1397},{"event": "Completed", "startTimeUtc": "2026-04-14T15:59:57.8586074Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T15:59:57.8540278Z","lastSend":"2026-04-14T15:59:57.8540278Z","lastReceive":"2026-04-14T15:59:57.8696512Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T15:59:57.8228604Z","DurationInMs":15.5454,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"5356f61f-3298-4205-a645-76f8f73bc119","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T15:59:57.8384058Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"075762d9-7ffc-479a-9294-15378c759cd7","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T15:59:57.8384058Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"0e247651-fc60-4af8-99a3-14444cb0b045","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718761,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"0e247651-fc60-4af8-99a3-14444cb0b045","ResponseTimeUtc":"2026-04-14T15:59:57.8696512Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 0e247651-fc60-4af8-99a3-14444cb0b045, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149358","BELatencyInMs":"0.711"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0023}]}
2026-04-14 16:00:15 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6270f8e6-c9cc-4806-81ba-911ee63853a3; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6270f8e6-c9cc-4806-81ba-911ee63853a3, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6270f8e6-c9cc-4806-81ba-911ee63853a3; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6270f8e6-c9cc-4806-81ba-911ee63853a3, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:00:15.682Z","duration in milliseconds":45.4269,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:00:15.5416567Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90905,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.052000000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":8.9709,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0041},{"name":"Read Collection","duration in milliseconds":8.7982,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:00:15.6823436Z","DurationInMs":15.638,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"4b5110db-1c7b-4e24-bb8d-00a346b35d05","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.214400000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.1754,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:59:21.4332683Z","cpu":2.262,"memory":5261876.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.094,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:31.4462762Z","cpu":1.248,"memory":5255532.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1012,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:41.4600527Z","cpu":2.652,"memory":5255096.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1164,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:51.4637978Z","cpu":1.641,"memory":5256664.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0972,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1},{"dateUtc":"2026-04-14T16:00:01.4686321Z","cpu":4.766,"memory":5213180.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0337,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:00:11.4804482Z","cpu":9.126,"memory":5185672.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0785,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.168,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.162800000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.141,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.1383,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.2419,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:00:15.6979816Z","EndTimeUTC":"2026-04-14T16:00:15.7135323Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:00:15.7291567Z","DurationInMs":15.624400000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"6270f8e6-c9cc-4806-81ba-911ee63853a3","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149358,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149358,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149358,"SessionToken":"-1#26149358","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.692","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:00:15 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:00:15.7135323Z", "durationInMs": 0.0094},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:00:15.7135417Z", "durationInMs": 18.8783},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:00:15.7324200Z", "durationInMs": 0.2576},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:00:15.7326776Z", "durationInMs": 2.0666},{"event": "Received", "startTimeUtc": "2026-04-14T16:00:15.7347442Z", "durationInMs": 0.1496},{"event": "Completed", "startTimeUtc": "2026-04-14T16:00:15.7348938Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:00:15.7291567Z","lastSend":"2026-04-14T16:00:15.7291567Z","lastReceive":"2026-04-14T16:00:15.7291567Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:00:15.6979816Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"8b829f29-bfdd-4fba-b63e-abdb8248711b","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:00:15.6979816Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"2c7b8145-bfe0-46cd-9d15-651332c4b4bb","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:00:15.6979816Z","DurationInMs":15.5507,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"6270f8e6-c9cc-4806-81ba-911ee63853a3","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1875,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"6270f8e6-c9cc-4806-81ba-911ee63853a3","ResponseTimeUtc":"2026-04-14T16:00:15.7291567Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 6270f8e6-c9cc-4806-81ba-911ee63853a3, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149358","BELatencyInMs":"0.692"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0023}]}
2026-04-14 16:00:32 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 15ecaeec-e35b-4cc4-9a6e-ff94ad23cb4a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 15ecaeec-e35b-4cc4-9a6e-ff94ad23cb4a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 15ecaeec-e35b-4cc4-9a6e-ff94ad23cb4a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 15ecaeec-e35b-4cc4-9a6e-ff94ad23cb4a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:00:32.251Z","duration in milliseconds":52.239900000000006,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:00:32.1104825Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90910,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0388},{"name":"Get PkValue From Stream","duration in milliseconds":7.4129000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":7.1691,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:00:32.2511736Z","DurationInMs":15.562600000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"b77d3556-a268-493b-a77b-1b06f837c761","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":44.5855,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":44.548,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T15:59:41.4600527Z","cpu":2.652,"memory":5255096.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1164,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T15:59:51.4637978Z","cpu":1.641,"memory":5256664.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0972,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1},{"dateUtc":"2026-04-14T16:00:01.4686321Z","cpu":4.766,"memory":5213180.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0337,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:00:11.4804482Z","cpu":9.126,"memory":5185672.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0785,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:00:21.4919763Z","cpu":2.418,"memory":5187808.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1145,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:00:31.5012006Z","cpu":2.109,"memory":5162000.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1146,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":44.5397,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":44.535000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":44.505300000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":44.502,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":43.5065,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:00:32.2823611Z","EndTimeUTC":"2026-04-14T16:00:32.2823611Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:00:32.2979994Z","DurationInMs":15.638300000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"15ecaeec-e35b-4cc4-9a6e-ff94ad23cb4a","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149358,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149358,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149358,"SessionToken":"-1#26149358","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.66","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:00:32 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:00:32.2823611Z", "durationInMs": 0.0088},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:00:32.2823699Z", "durationInMs": 19.1126},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:00:32.3014825Z", "durationInMs": 0.1712},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:00:32.3016537Z", "durationInMs": 1.8543},{"event": "Received", "startTimeUtc": "2026-04-14T16:00:32.3035080Z", "durationInMs": 0.106},{"event": "Completed", "startTimeUtc": "2026-04-14T16:00:32.3036140Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:00:32.2979994Z","lastSend":"2026-04-14T16:00:32.2979994Z","lastReceive":"2026-04-14T16:00:32.2979994Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:00:32.2667362Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"ef4f3bbc-d6ae-42ea-ad42-a1fbe3a6e51d","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:00:32.2667362Z","DurationInMs":15.6249,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"3e2414e9-8d5d-4475-88f2-66eb16c127cb","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:00:32.2823611Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"15ecaeec-e35b-4cc4-9a6e-ff94ad23cb4a","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1875169,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"15ecaeec-e35b-4cc4-9a6e-ff94ad23cb4a","ResponseTimeUtc":"2026-04-14T16:00:32.2979994Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 15ecaeec-e35b-4cc4-9a6e-ff94ad23cb4a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149358","BELatencyInMs":"0.66"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 16:00:52 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 51f0a24b-6a11-42f1-98fc-201954a606bc; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 51f0a24b-6a11-42f1-98fc-201954a606bc, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 51f0a24b-6a11-42f1-98fc-201954a606bc; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 51f0a24b-6a11-42f1-98fc-201954a606bc, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:00:52.052Z","duration in milliseconds":47.1471,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:00:51.9271169Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90914,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0473},{"name":"Get PkValue From Stream","duration in milliseconds":6.2561,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0041},{"name":"Read Collection","duration in milliseconds":6.0947000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:00:52.0522982Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"d5888b61-ae84-4933-8b86-664041763ece","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":40.6622,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":40.6287,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:00:01.4686321Z","cpu":4.766,"memory":5213180.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0337,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:00:11.4804482Z","cpu":9.126,"memory":5185672.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0785,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:00:21.4919763Z","cpu":2.418,"memory":5187808.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1145,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:00:31.5012006Z","cpu":2.109,"memory":5162000.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1146,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:00:41.5156653Z","cpu":2.730,"memory":5161916.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1181,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:00:51.5298156Z","cpu":2.574,"memory":5160488.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0919,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":40.620200000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":40.615,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":40.5948,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":40.590900000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":39.743700000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:00:52.0678238Z","EndTimeUTC":"2026-04-14T16:00:52.0678238Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:00:52.0989929Z","DurationInMs":31.1691,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"51f0a24b-6a11-42f1-98fc-201954a606bc","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149358,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149358,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149358,"SessionToken":"-1#26149358","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.817","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:00:52 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:00:52.0678238Z", "durationInMs": 0.0091},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:00:52.0678329Z", "durationInMs": 20.6591},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:00:52.0884920Z", "durationInMs": 0.1947},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:00:52.0886867Z", "durationInMs": 2.4264},{"event": "Received", "startTimeUtc": "2026-04-14T16:00:52.0911131Z", "durationInMs": 0.139},{"event": "Completed", "startTimeUtc": "2026-04-14T16:00:52.0912521Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:00:52.0989929Z","lastSend":"2026-04-14T16:00:52.0989929Z","lastReceive":"2026-04-14T16:00:52.0989929Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:00:52.0522982Z","DurationInMs":15.5256,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"de867671-dd24-473f-b5b7-1dbd1a087fa2","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:00:52.0678238Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"46d2a749-3978-49df-b21b-f7e575ce8d61","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:00:52.0678238Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"51f0a24b-6a11-42f1-98fc-201954a606bc","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.171876,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"51f0a24b-6a11-42f1-98fc-201954a606bc","ResponseTimeUtc":"2026-04-14T16:00:52.0989929Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 51f0a24b-6a11-42f1-98fc-201954a606bc, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149358","BELatencyInMs":"0.817"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 16:07:25 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 7e3d35d6-7a7a-482f-b20f-06c4ca912560; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 7e3d35d6-7a7a-482f-b20f-06c4ca912560, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 7e3d35d6-7a7a-482f-b20f-06c4ca912560; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 7e3d35d6-7a7a-482f-b20f-06c4ca912560, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:07:25.616Z","duration in milliseconds":75.4358,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:07:25.1316957Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90946,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.042},{"name":"Get PkValue From Stream","duration in milliseconds":6.8173,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0034000000000000002},{"name":"Read Collection","duration in milliseconds":6.6736,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:07:25.6168381Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"22c42410-2765-4606-b99a-e9dd0a7b3d8a","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":68.329000000000008,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":68.2928,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:06:31.7830146Z","cpu":1.172,"memory":5280148.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0994,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:06:41.8001161Z","cpu":0.468,"memory":5279580.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2872,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:06:51.8043350Z","cpu":1.094,"memory":5279132.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2549,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:07:01.8201773Z","cpu":0.702,"memory":5278640.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1166,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:07:11.8304033Z","cpu":1.092,"memory":5279448.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1116,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:07:21.8348399Z","cpu":0.625,"memory":5282064.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1018,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":68.280200000000008,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":68.2756,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":68.25030000000001,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":68.2463,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":66.9269,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:07:25.6636642Z","EndTimeUTC":"2026-04-14T16:07:25.6636642Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:07:25.6949179Z","DurationInMs":31.253700000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"7e3d35d6-7a7a-482f-b20f-06c4ca912560","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149358,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149358,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149358,"SessionToken":"-1#26149358","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.688","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:07:25 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:07:25.6636642Z", "durationInMs": 0.0094},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:07:25.6636736Z", "durationInMs": 19.9741},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:07:25.6836477Z", "durationInMs": 0.2002},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:07:25.6838479Z", "durationInMs": 3.2695},{"event": "Received", "startTimeUtc": "2026-04-14T16:07:25.6871174Z", "durationInMs": 0.2245},{"event": "Completed", "startTimeUtc": "2026-04-14T16:07:25.6873419Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:07:25.6792849Z","lastSend":"2026-04-14T16:07:25.6792849Z","lastReceive":"2026-04-14T16:07:25.6792849Z"},"requestSizeInBytes":1608,"requestBodySizeInBytes":1156,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:07:25.6168381Z","DurationInMs":15.571100000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a0f0b1de-892d-4d10-a180-212f45fcaba1","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:07:25.6324092Z","DurationInMs":15.6981,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"369db4c5-f261-4b5e-a8ab-78b478d2de85","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:07:25.6636642Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"7e3d35d6-7a7a-482f-b20f-06c4ca912560","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.5632222,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"7e3d35d6-7a7a-482f-b20f-06c4ca912560","ResponseTimeUtc":"2026-04-14T16:07:25.6949179Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 7e3d35d6-7a7a-482f-b20f-06c4ca912560, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149358","BELatencyInMs":"0.688"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.003}]}
2026-04-14 16:14:15 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 38feb74e-d480-41e3-8797-56e6696283e7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 38feb74e-d480-41e3-8797-56e6696283e7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 38feb74e-d480-41e3-8797-56e6696283e7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 38feb74e-d480-41e3-8797-56e6696283e7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:14:15.081Z","duration in milliseconds":65.717700000000008,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:14:14.2064597Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90965,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0499},{"name":"Get PkValue From Stream","duration in milliseconds":11.384300000000001,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014},{"name":"Read Collection","duration in milliseconds":11.1768,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:14:15.0815552Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"27c40ac3-7b71-4d21-bea9-79b0d47c7a36","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":54.0914,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":54.051500000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:13:22.1565437Z","cpu":1.484,"memory":5294184.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1175,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:13:32.1620208Z","cpu":3.354,"memory":5261740.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1035,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:13:42.1694619Z","cpu":1.172,"memory":5262056.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2293,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:13:52.1810443Z","cpu":1.092,"memory":5274388.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1113,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:14:02.1851646Z","cpu":1.641,"memory":5272376.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2614,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:14:12.1908139Z","cpu":0.781,"memory":5271220.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1074,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":54.0433,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":54.0392,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":54.0139,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":54.0108,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":53.0032,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:14:15.0970991Z","EndTimeUTC":"2026-04-14T16:14:15.0970991Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:14:15.1439786Z","DurationInMs":46.8795,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"38feb74e-d480-41e3-8797-56e6696283e7","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.758","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:14:15 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:14:15.0970991Z", "durationInMs": 0.0086},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:14:15.0971077Z", "durationInMs": 35.4257},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:14:15.1325334Z", "durationInMs": 0.2876},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:14:15.1328210Z", "durationInMs": 1.9668},{"event": "Received", "startTimeUtc": "2026-04-14T16:14:15.1347878Z", "durationInMs": 0.1486},{"event": "Completed", "startTimeUtc": "2026-04-14T16:14:15.1349364Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:14:15.1439786Z","lastSend":"2026-04-14T16:14:15.1439786Z","lastReceive":"2026-04-14T16:14:15.1439786Z"},"requestSizeInBytes":1623,"requestBodySizeInBytes":1171,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:14:15.0970991Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"40dd72e0-58d5-4b68-b50e-9f7810b2101c","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:14:15.0970991Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"ccbb9c12-29b6-4512-9a97-e77cb1393871","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:14:15.0970991Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"38feb74e-d480-41e3-8797-56e6696283e7","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.93751889999999993,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"38feb74e-d480-41e3-8797-56e6696283e7","ResponseTimeUtc":"2026-04-14T16:14:15.1439786Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 38feb74e-d480-41e3-8797-56e6696283e7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.758"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 16:14:36 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 61c3b94f-19fa-45f1-b967-6856fb6cde83; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 61c3b94f-19fa-45f1-b967-6856fb6cde83, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 61c3b94f-19fa-45f1-b967-6856fb6cde83; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 61c3b94f-19fa-45f1-b967-6856fb6cde83, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:14:35.978Z","duration in milliseconds":48.2254,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:14:35.8844894Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90968,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.04},{"name":"Get PkValue From Stream","duration in milliseconds":7.5710000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0033},{"name":"Read Collection","duration in milliseconds":7.4289000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:14:35.9782399Z","DurationInMs":15.6972,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"85ef7c56-a2c8-454b-b70a-92dc42686fe0","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":40.3783,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":40.3527,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:13:42.1694619Z","cpu":1.172,"memory":5262056.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2293,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:13:52.1810443Z","cpu":1.092,"memory":5274388.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1113,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:14:02.1851646Z","cpu":1.641,"memory":5272376.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2614,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:14:12.1908139Z","cpu":0.781,"memory":5271220.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1074,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:14:22.1961365Z","cpu":2.340,"memory":5286996.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0929,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:14:32.1970037Z","cpu":2.109,"memory":5277140.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.289,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":40.3375,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":40.3325,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":40.3104,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":40.305600000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":39.1276,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:14:36.0094907Z","EndTimeUTC":"2026-04-14T16:14:36.0094907Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:14:36.0251172Z","DurationInMs":15.6265,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"61c3b94f-19fa-45f1-b967-6856fb6cde83","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.584","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:14:36 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:14:36.0094907Z", "durationInMs": 0.0093},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:14:36.0095000Z", "durationInMs": 18.7308},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:14:36.0282308Z", "durationInMs": 0.2216},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:14:36.0284524Z", "durationInMs": 2.1491},{"event": "Received", "startTimeUtc": "2026-04-14T16:14:36.0306015Z", "durationInMs": 0.1316},{"event": "Completed", "startTimeUtc": "2026-04-14T16:14:36.0307331Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:14:36.0251172Z","lastSend":"2026-04-14T16:14:36.0251172Z","lastReceive":"2026-04-14T16:14:36.0251172Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:14:35.9939371Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"f3e4e86a-3bac-4aa3-bedd-11857a2f6830","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:14:35.9939371Z","DurationInMs":15.553600000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0956e31a-0cb7-45db-ade5-f858b78a42de","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:14:36.0094907Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"61c3b94f-19fa-45f1-b967-6856fb6cde83","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1406278,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"61c3b94f-19fa-45f1-b967-6856fb6cde83","ResponseTimeUtc":"2026-04-14T16:14:36.0251172Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 61c3b94f-19fa-45f1-b967-6856fb6cde83, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.584"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 16:15:23 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 33ab0b37-9eb0-4b93-a3af-809b740cb809; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 33ab0b37-9eb0-4b93-a3af-809b740cb809, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 33ab0b37-9eb0-4b93-a3af-809b740cb809; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 33ab0b37-9eb0-4b93-a3af-809b740cb809, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:15:23.796Z","duration in milliseconds":65.8095,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:15:23.6869871Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90972,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0531},{"name":"Get PkValue From Stream","duration in milliseconds":6.7486000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002},{"name":"Read Collection","duration in milliseconds":6.5922,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:15:23.7964364Z","DurationInMs":15.6088,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"0e83ec72-7109-4116-8ec4-e311cfee73b9","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":58.8175,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":58.783300000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:14:32.1970037Z","cpu":2.109,"memory":5277140.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.289,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:14:42.2063085Z","cpu":1.719,"memory":5271264.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1101,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:14:52.2096938Z","cpu":1.638,"memory":5269692.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.103,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:15:02.2188739Z","cpu":6.719,"memory":5191732.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1246,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:15:12.2348473Z","cpu":3.666,"memory":5236336.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1954,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:15:22.2495902Z","cpu":0.936,"memory":5221952.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1787,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":58.7744,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":58.770500000000006,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":58.748900000000006,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":58.7462,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":57.8982,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:15:23.8120452Z","EndTimeUTC":"2026-04-14T16:15:23.8276821Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:15:23.8589660Z","DurationInMs":31.283900000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"33ab0b37-9eb0-4b93-a3af-809b740cb809","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.666","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:15:23 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:15:23.8276821Z", "durationInMs": 0.0096},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:15:23.8276917Z", "durationInMs": 39.6424},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:15:23.8673341Z", "durationInMs": 0.2105},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:15:23.8675446Z", "durationInMs": 1.7683},{"event": "Received", "startTimeUtc": "2026-04-14T16:15:23.8693129Z", "durationInMs": 0.1431},{"event": "Completed", "startTimeUtc": "2026-04-14T16:15:23.8694560Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:15:23.8589660Z","lastSend":"2026-04-14T16:15:23.8589660Z","lastReceive":"2026-04-14T16:15:23.8589660Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:15:23.8120452Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"bedcb05e-205c-4202-8e98-8d31992ff98f","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:15:23.8120452Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"df5dcd41-22d3-4c69-ad6e-d57300cf5ebe","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:15:23.8120452Z","DurationInMs":15.6369,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"33ab0b37-9eb0-4b93-a3af-809b740cb809","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1719789,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"33ab0b37-9eb0-4b93-a3af-809b740cb809","ResponseTimeUtc":"2026-04-14T16:15:23.8589660Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 33ab0b37-9eb0-4b93-a3af-809b740cb809, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.666"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0025}]}
2026-04-14 16:15:42 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5a7ea213-fa83-4fa1-8e1e-5a207549ebb5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5a7ea213-fa83-4fa1-8e1e-5a207549ebb5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5a7ea213-fa83-4fa1-8e1e-5a207549ebb5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5a7ea213-fa83-4fa1-8e1e-5a207549ebb5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:15:42.037Z","duration in milliseconds":43.0167,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:15:41.9127277Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90974,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.049600000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":6.7494000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0015},{"name":"Read Collection","duration in milliseconds":6.575,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:15:42.0377470Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"7d2eff73-84e3-483f-b3bb-66879c8952df","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.028800000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.991,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:14:42.2063085Z","cpu":1.719,"memory":5271264.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1101,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:14:52.2096938Z","cpu":1.638,"memory":5269692.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.103,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:15:02.2188739Z","cpu":6.719,"memory":5191732.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1246,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:15:12.2348473Z","cpu":3.666,"memory":5236336.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1954,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:15:22.2495902Z","cpu":0.936,"memory":5221952.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1787,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:15:32.2599756Z","cpu":0.936,"memory":5224256.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1279,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.9824,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.979,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.9596,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.956,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.8619,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:15:42.0534126Z","EndTimeUTC":"2026-04-14T16:15:42.0534126Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:15:42.0689793Z","DurationInMs":15.5667,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"5a7ea213-fa83-4fa1-8e1e-5a207549ebb5","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.539","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:15:42 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:15:42.0534126Z", "durationInMs": 0.0096},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:15:42.0534222Z", "durationInMs": 16.8434},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:15:42.0702656Z", "durationInMs": 0.2116},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:15:42.0704772Z", "durationInMs": 2.0035},{"event": "Received", "startTimeUtc": "2026-04-14T16:15:42.0724807Z", "durationInMs": 0.1478},{"event": "Completed", "startTimeUtc": "2026-04-14T16:15:42.0726285Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:15:42.0689793Z","lastSend":"2026-04-14T16:15:42.0689793Z","lastReceive":"2026-04-14T16:15:42.0689793Z"},"requestSizeInBytes":1638,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:15:42.0377470Z","DurationInMs":15.665600000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"15151502-5c13-477e-b5d1-419576c0411f","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:15:42.0534126Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"65c4c3b1-d460-4728-9a89-950948d26fc4","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:15:42.0534126Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"5a7ea213-fa83-4fa1-8e1e-5a207549ebb5","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1562516,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"5a7ea213-fa83-4fa1-8e1e-5a207549ebb5","ResponseTimeUtc":"2026-04-14T16:15:42.0846083Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 5a7ea213-fa83-4fa1-8e1e-5a207549ebb5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.539"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001}]}
2026-04-14 16:16:03 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: a64a46b6-5e51-49b3-ab58-b856a3cc2e92; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: a64a46b6-5e51-49b3-ab58-b856a3cc2e92, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: a64a46b6-5e51-49b3-ab58-b856a3cc2e92; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: a64a46b6-5e51-49b3-ab58-b856a3cc2e92, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:16:02.969Z","duration in milliseconds":44.9074,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:16:02.8440572Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90976,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0645},{"name":"Get PkValue From Stream","duration in milliseconds":5.78,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.001},{"name":"Read Collection","duration in milliseconds":5.605,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:16:02.9690591Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"a203cc80-4435-4051-ba2e-a7328f5c9dff","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":38.807500000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":38.7714,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:15:12.2348473Z","cpu":3.666,"memory":5236336.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1954,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:15:22.2495902Z","cpu":0.936,"memory":5221952.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1787,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:15:32.2599756Z","cpu":0.936,"memory":5224256.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1279,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:15:42.2721200Z","cpu":1.716,"memory":5227700.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0963,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:15:52.2880320Z","cpu":1.170,"memory":5223684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1024,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:16:02.2972502Z","cpu":2.109,"memory":5185444.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0919,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":38.763400000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":38.757400000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":38.7338,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":38.731300000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":37.9037,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:16:02.9846839Z","EndTimeUTC":"2026-04-14T16:16:02.9846839Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:16:03.0159410Z","DurationInMs":31.2571,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"a64a46b6-5e51-49b3-ab58-b856a3cc2e92","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.651","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:16:03 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:16:02.9846839Z", "durationInMs": 0.0109},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:16:02.9846948Z", "durationInMs": 19.9954},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:16:03.0046902Z", "durationInMs": 0.2288},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:16:03.0049190Z", "durationInMs": 1.9106},{"event": "Received", "startTimeUtc": "2026-04-14T16:16:03.0068296Z", "durationInMs": 0.1175},{"event": "Completed", "startTimeUtc": "2026-04-14T16:16:03.0069471Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:16:03.0159410Z","lastSend":"2026-04-14T16:16:03.0159410Z","lastReceive":"2026-04-14T16:16:03.0159410Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:16:02.9690591Z","DurationInMs":15.6248,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"697f9d88-a2b4-4d53-81df-da3569a3542f","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:16:02.9846839Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"1177411e-5882-44b1-a157-f8c5cf3cd40d","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:16:02.9846839Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"a64a46b6-5e51-49b3-ab58-b856a3cc2e92","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718838,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"a64a46b6-5e51-49b3-ab58-b856a3cc2e92","ResponseTimeUtc":"2026-04-14T16:16:03.0159410Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: a64a46b6-5e51-49b3-ab58-b856a3cc2e92, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.651"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 16:16:27 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 1acfa954-970c-4930-8201-aee2881ad6e3; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 1acfa954-970c-4930-8201-aee2881ad6e3, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 1acfa954-970c-4930-8201-aee2881ad6e3; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 1acfa954-970c-4930-8201-aee2881ad6e3, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:16:27.094Z","duration in milliseconds":42.6433,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:16:26.9225055Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90978,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0497},{"name":"Get PkValue From Stream","duration in milliseconds":7.2351,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0036000000000000003},{"name":"Read Collection","duration in milliseconds":7.0819,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:16:27.0944534Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"8ae5d7df-916e-40e2-8ad0-9849ae4093f9","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.1822,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.1495,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:15:32.2599756Z","cpu":0.936,"memory":5224256.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1279,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:15:42.2721200Z","cpu":1.716,"memory":5227700.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0963,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:15:52.2880320Z","cpu":1.170,"memory":5223684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1024,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:16:02.2972502Z","cpu":2.109,"memory":5185444.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0919,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:16:12.2995526Z","cpu":1.797,"memory":5197524.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0963,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:16:22.3143515Z","cpu":1.794,"memory":5201684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.135,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.141,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.1379,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.118700000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.1165,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.2648,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:16:27.1100565Z","EndTimeUTC":"2026-04-14T16:16:27.1100565Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:16:27.1256282Z","DurationInMs":15.5717,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"1acfa954-970c-4930-8201-aee2881ad6e3","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.578","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:16:27 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:16:27.1100565Z", "durationInMs": 0.0101},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:16:27.1100666Z", "durationInMs": 17.9336},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:16:27.1280002Z", "durationInMs": 0.1908},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:16:27.1281910Z", "durationInMs": 1.7397},{"event": "Received", "startTimeUtc": "2026-04-14T16:16:27.1299307Z", "durationInMs": 0.1326},{"event": "Completed", "startTimeUtc": "2026-04-14T16:16:27.1300633Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:16:27.1256282Z","lastSend":"2026-04-14T16:16:27.1256282Z","lastReceive":"2026-04-14T16:16:27.1256282Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:16:27.0944534Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"8ea30289-99b2-48b3-b24f-4400ba78dc52","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:16:27.0944534Z","DurationInMs":15.603100000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"fa6f75c1-fd5f-4ac0-8cc1-4d6f8f8c3da2","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:16:27.1100565Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"1acfa954-970c-4930-8201-aee2881ad6e3","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2031227,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"1acfa954-970c-4930-8201-aee2881ad6e3","ResponseTimeUtc":"2026-04-14T16:16:27.1256282Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 1acfa954-970c-4930-8201-aee2881ad6e3, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.578"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 16:17:12 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: f7030ad1-780d-49e5-b18a-46e430eac8f0; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: f7030ad1-780d-49e5-b18a-46e430eac8f0, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: f7030ad1-780d-49e5-b18a-46e430eac8f0; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: f7030ad1-780d-49e5-b18a-46e430eac8f0, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:17:11.949Z","duration in milliseconds":76.192100000000011,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:17:11.5118383Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90981,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.043500000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":8.1211,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.00090000000000000008},{"name":"Read Collection","duration in milliseconds":7.9861,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:17:11.9493495Z","DurationInMs":15.6917,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"675a738c-d13c-44ee-ae92-81ec26f10c32","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":67.861,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":67.8305,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:16:12.2995526Z","cpu":1.797,"memory":5197524.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0963,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:16:22.3143515Z","cpu":1.794,"memory":5201684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.135,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:16:32.3168694Z","cpu":1.094,"memory":5203612.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.104,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:16:42.3259372Z","cpu":1.872,"memory":5194636.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1087,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:16:52.3359937Z","cpu":0.390,"memory":5194476.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1002,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:17:02.3503506Z","cpu":1.094,"memory":5193892.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1032,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":67.8236,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":67.8201,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":67.7981,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":67.7954,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":66.927,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:17:11.9963103Z","EndTimeUTC":"2026-04-14T16:17:11.9963103Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:17:12.0275564Z","DurationInMs":31.246100000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"f7030ad1-780d-49e5-b18a-46e430eac8f0","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.554","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:17:12 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:17:11.9963103Z", "durationInMs": 0.0085},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:17:11.9963188Z", "durationInMs": 30.2797},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:17:12.0265985Z", "durationInMs": 0.2266},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:17:12.0268251Z", "durationInMs": 1.6452},{"event": "Received", "startTimeUtc": "2026-04-14T16:17:12.0284703Z", "durationInMs": 0.144},{"event": "Completed", "startTimeUtc": "2026-04-14T16:17:12.0286143Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:17:12.0275564Z","lastSend":"2026-04-14T16:17:12.0275564Z","lastReceive":"2026-04-14T16:17:12.0275564Z"},"requestSizeInBytes":1606,"requestBodySizeInBytes":1156,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:17:11.9650412Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a352b6c5-d613-4020-8899-d0a0f0b2d3ca","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:17:11.9650412Z","DurationInMs":31.2691,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"5e19328c-c531-4f1f-9a00-3d08bee02e95","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:17:11.9963103Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"f7030ad1-780d-49e5-b18a-46e430eac8f0","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.5157181,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"f7030ad1-780d-49e5-b18a-46e430eac8f0","ResponseTimeUtc":"2026-04-14T16:17:12.0275564Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: f7030ad1-780d-49e5-b18a-46e430eac8f0, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.554"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 16:18:05 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 82de8eab-0235-4271-90fb-1553b998ba86; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 82de8eab-0235-4271-90fb-1553b998ba86, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 82de8eab-0235-4271-90fb-1553b998ba86; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 82de8eab-0235-4271-90fb-1553b998ba86, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:18:05.581Z","duration in milliseconds":46.3462,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:18:05.1592092Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90982,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0482},{"name":"Get PkValue From Stream","duration in milliseconds":6.3573,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014},{"name":"Read Collection","duration in milliseconds":6.1994000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:18:05.5811489Z","DurationInMs":15.6362,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"2eb50904-2c68-4df9-ac0d-a744f7525add","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":39.7526,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":39.7167,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:17:12.3556218Z","cpu":2.106,"memory":5194128.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1194,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:17:22.3682719Z","cpu":1.326,"memory":5193480.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1025,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:17:32.3716533Z","cpu":0.547,"memory":5199456.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0566,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:17:42.3842071Z","cpu":1.170,"memory":5195588.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2773,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:17:52.3954890Z","cpu":1.250,"memory":5208980.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0932,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:18:02.3990870Z","cpu":0.781,"memory":5260868.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0984,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":39.7083,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":39.704100000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":39.6828,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":39.6803,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":38.7276,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:18:05.5967851Z","EndTimeUTC":"2026-04-14T16:18:05.6123426Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:18:05.6280771Z","DurationInMs":15.7345,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"82de8eab-0235-4271-90fb-1553b998ba86","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.795","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:18:05 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:18:05.6123426Z", "durationInMs": 0.009},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:18:05.6123516Z", "durationInMs": 21.9328},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:18:05.6342844Z", "durationInMs": 0.2233},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:18:05.6345077Z", "durationInMs": 1.826},{"event": "Received", "startTimeUtc": "2026-04-14T16:18:05.6363337Z", "durationInMs": 0.1417},{"event": "Completed", "startTimeUtc": "2026-04-14T16:18:05.6364754Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:18:05.6280771Z","lastSend":"2026-04-14T16:18:05.6280771Z","lastReceive":"2026-04-14T16:18:05.6280771Z"},"requestSizeInBytes":1625,"requestBodySizeInBytes":1171,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:18:05.5967851Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"e6f8ad84-f55c-4229-b0a4-b51b8b09f934","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:18:05.5967851Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"5768c3f3-c810-4e28-bc4d-65e35c80bcb1","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:18:05.5967851Z","DurationInMs":15.557500000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"82de8eab-0235-4271-90fb-1553b998ba86","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.48448789999999997,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"82de8eab-0235-4271-90fb-1553b998ba86","ResponseTimeUtc":"2026-04-14T16:18:05.6280771Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 82de8eab-0235-4271-90fb-1553b998ba86, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.795"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 16:20:18 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c43b392d-234f-44b5-8e67-213c1770c998; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c43b392d-234f-44b5-8e67-213c1770c998, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c43b392d-234f-44b5-8e67-213c1770c998; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c43b392d-234f-44b5-8e67-213c1770c998, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:20:18.269Z","duration in milliseconds":53.268,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:20:18.1596912Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":90992,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.042},{"name":"Get PkValue From Stream","duration in milliseconds":7.7225,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0036000000000000003},{"name":"Read Collection","duration in milliseconds":7.5695000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:20:18.2691498Z","DurationInMs":15.545100000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"5c391bcf-35b0-4712-932e-ff9c3022a4ca","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":45.3273,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":45.2929,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:19:22.4751358Z","cpu":2.656,"memory":5261872.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0977,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:19:32.4888797Z","cpu":4.056,"memory":5253320.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1078,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:19:42.5049824Z","cpu":1.248,"memory":5254360.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1052,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:19:52.5092488Z","cpu":0.859,"memory":5255192.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1513,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:20:02.5150765Z","cpu":1.638,"memory":5239492.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1005,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:20:12.5263922Z","cpu":2.578,"memory":5206172.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1096,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":45.2855,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":45.282000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":45.260200000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":45.2575,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":44.3845,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:20:18.3003265Z","EndTimeUTC":"2026-04-14T16:20:18.3003265Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:20:18.3315691Z","DurationInMs":31.242600000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c43b392d-234f-44b5-8e67-213c1770c998","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.805","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:20:18 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:20:18.3003265Z", "durationInMs": 0.0096},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:20:18.3003361Z", "durationInMs": 17.8836},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:20:18.3182197Z", "durationInMs": 0.193},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:20:18.3184127Z", "durationInMs": 2.31},{"event": "Received", "startTimeUtc": "2026-04-14T16:20:18.3207227Z", "durationInMs": 0.1454},{"event": "Completed", "startTimeUtc": "2026-04-14T16:20:18.3208681Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:20:18.3159456Z","lastSend":"2026-04-14T16:20:18.3159456Z","lastReceive":"2026-04-14T16:20:18.3315691Z"},"requestSizeInBytes":1629,"requestBodySizeInBytes":1181,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:20:18.2846949Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0c6ca3a2-d6dc-4433-bc1f-50b135bef2d2","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:20:18.2846949Z","DurationInMs":15.6316,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"bb34f230-8689-4daf-b107-975d1763138d","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:20:18.3003265Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"c43b392d-234f-44b5-8e67-213c1770c998","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718779,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c43b392d-234f-44b5-8e67-213c1770c998","ResponseTimeUtc":"2026-04-14T16:20:18.3315691Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c43b392d-234f-44b5-8e67-213c1770c998, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.805"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 16:20:53 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 81d13a37-64d9-4fcc-ae75-101ba73a0a26; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 81d13a37-64d9-4fcc-ae75-101ba73a0a26, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 81d13a37-64d9-4fcc-ae75-101ba73a0a26; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 81d13a37-64d9-4fcc-ae75-101ba73a0a26, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:20:53.187Z","duration in milliseconds":4.1705000000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:20:52.7973044Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91034,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0482},{"name":"Get PkValue From Stream","duration in milliseconds":0.16740000000000002,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":3.7954000000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":3.7689000000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:20:02.5150765Z","cpu":1.638,"memory":5239492.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1005,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:20:12.5263922Z","cpu":2.578,"memory":5206172.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1096,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:20:22.5311099Z","cpu":2.730,"memory":5209284.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0975,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:20:32.5339498Z","cpu":2.031,"memory":5209992.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0664,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:20:42.5376152Z","cpu":8.906,"memory":5170704.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1029,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:20:52.5474338Z","cpu":3.588,"memory":5168060.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1114,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":3.7644,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":3.7615000000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":3.7454,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":3.7436000000000003,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":2.7865,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:20:53.1879371Z","DurationInMs":0,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"81d13a37-64d9-4fcc-ae75-101ba73a0a26","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.753","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:20:53 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:20:53.1879371Z", "durationInMs": 0.0035},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:20:53.1879406Z", "durationInMs": 0.0043},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:20:53.1879449Z", "durationInMs": 0.2076},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:20:53.1881525Z", "durationInMs": 1.8046},{"event": "Received", "startTimeUtc": "2026-04-14T16:20:53.1899571Z", "durationInMs": 0.0685},{"event": "Completed", "startTimeUtc": "2026-04-14T16:20:53.1900256Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:20:53.1879371Z","lastSend":"2026-04-14T16:20:53.1879371Z","lastReceive":"2026-04-14T16:20:53.1879371Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1186,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.3906327,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"81d13a37-64d9-4fcc-ae75-101ba73a0a26","ResponseTimeUtc":"2026-04-14T16:20:53.1879371Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 81d13a37-64d9-4fcc-ae75-101ba73a0a26, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.753"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0011}]}
2026-04-14 16:22:18 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5060f832-ce75-4581-9055-9be7ccc7ad15; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5060f832-ce75-4581-9055-9be7ccc7ad15, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5060f832-ce75-4581-9055-9be7ccc7ad15; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5060f832-ce75-4581-9055-9be7ccc7ad15, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:22:18.878Z","duration in milliseconds":41.813500000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:22:18.8003567Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91045,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0505},{"name":"Get PkValue From Stream","duration in milliseconds":6.3341,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0019},{"name":"Read Collection","duration in milliseconds":6.1739000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:22:18.8784836Z","DurationInMs":15.6934,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"50a5e1a9-62e6-453a-907e-d808cedb336a","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.252700000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.2175,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:21:22.5704935Z","cpu":1.406,"memory":5243424.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1417,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:21:32.5732600Z","cpu":0.859,"memory":5242684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1105,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:21:42.5799960Z","cpu":0.624,"memory":5238412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1395,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:21:52.5882512Z","cpu":1.172,"memory":5244872.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1022,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:22:02.6021714Z","cpu":1.248,"memory":5243384.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2346,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:22:12.6173454Z","cpu":1.014,"memory":5240184.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0942,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.211200000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.2061,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.186,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.182900000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.3078,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:22:18.8941770Z","EndTimeUTC":"2026-04-14T16:22:18.8941770Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:22:18.9253585Z","DurationInMs":31.1815,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"5060f832-ce75-4581-9055-9be7ccc7ad15","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.641","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:22:18 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:22:18.8941770Z", "durationInMs": 0.0097},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:22:18.8941867Z", "durationInMs": 18.4304},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:22:18.9126171Z", "durationInMs": 0.2038},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:22:18.9128209Z", "durationInMs": 1.8556},{"event": "Received", "startTimeUtc": "2026-04-14T16:22:18.9146765Z", "durationInMs": 0.1368},{"event": "Completed", "startTimeUtc": "2026-04-14T16:22:18.9148133Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:22:18.9097394Z","lastSend":"2026-04-14T16:22:18.9097394Z","lastReceive":"2026-04-14T16:22:18.9253585Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:22:18.8941770Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"fb991882-27a5-4117-942c-6350c68a05d4","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:22:18.8941770Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"bf68f00b-346c-4add-9341-a0a14122c5e9","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:22:18.8941770Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"5060f832-ce75-4581-9055-9be7ccc7ad15","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1250018,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"5060f832-ce75-4581-9055-9be7ccc7ad15","ResponseTimeUtc":"2026-04-14T16:22:18.9253585Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 5060f832-ce75-4581-9055-9be7ccc7ad15, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.641"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 16:23:25 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: ca22198e-ce3c-43c9-90a7-f29b21a9e162; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: ca22198e-ce3c-43c9-90a7-f29b21a9e162, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: ca22198e-ce3c-43c9-90a7-f29b21a9e162; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: ca22198e-ce3c-43c9-90a7-f29b21a9e162, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:23:25.151Z","duration in milliseconds":48.92,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:23:24.9951411Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91059,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0444},{"name":"Get PkValue From Stream","duration in milliseconds":6.2981000000000007,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002},{"name":"Read Collection","duration in milliseconds":6.138,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:23:25.1513960Z","DurationInMs":15.6254,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"0e701038-458d-4c85-bffc-f3b8dd67e15e","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":42.3004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":42.264700000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:22:32.6378070Z","cpu":1.250,"memory":5257212.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.062,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:22:42.6528406Z","cpu":3.432,"memory":5253056.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0685,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:22:52.6615523Z","cpu":0.780,"memory":5251756.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2547,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:02.6764396Z","cpu":2.730,"memory":5355140.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1291,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:12.6863124Z","cpu":1.797,"memory":5348488.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0923,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:22.6975887Z","cpu":1.716,"memory":5343112.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1026,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":42.2573,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":42.2533,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":42.231100000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":42.2282,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":41.241600000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:23:25.1670214Z","EndTimeUTC":"2026-04-14T16:23:25.1826482Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:23:25.1983372Z","DurationInMs":15.689,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"ca22198e-ce3c-43c9-90a7-f29b21a9e162","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.692","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:23:25 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:23:25.1826482Z", "durationInMs": 0.0078},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:23:25.1826560Z", "durationInMs": 18.4781},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:23:25.2011341Z", "durationInMs": 0.2115},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:23:25.2013456Z", "durationInMs": 6.7481},{"event": "Received", "startTimeUtc": "2026-04-14T16:23:25.2080937Z", "durationInMs": 0.1692},{"event": "Completed", "startTimeUtc": "2026-04-14T16:23:25.2082629Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:23:25.1983372Z","lastSend":"2026-04-14T16:23:25.1983372Z","lastReceive":"2026-04-14T16:23:25.1983372Z"},"requestSizeInBytes":1630,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:23:25.1670214Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a7119750-c167-47ae-8fc8-9ee14de308cd","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:23:25.1670214Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"bae442ab-e725-4fbf-91d2-9ee78cbc85ac","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:23:25.1670214Z","DurationInMs":15.626800000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"ca22198e-ce3c-43c9-90a7-f29b21a9e162","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2031961,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"ca22198e-ce3c-43c9-90a7-f29b21a9e162","ResponseTimeUtc":"2026-04-14T16:23:25.1983372Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: ca22198e-ce3c-43c9-90a7-f29b21a9e162, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.692"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 16:23:49 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 03ce06b4-e7b2-41e3-bebf-23a8d9c6eacb; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 03ce06b4-e7b2-41e3-bebf-23a8d9c6eacb, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 03ce06b4-e7b2-41e3-bebf-23a8d9c6eacb; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 03ce06b4-e7b2-41e3-bebf-23a8d9c6eacb, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:23:49.812Z","duration in milliseconds":47.966,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:23:49.6878260Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91064,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.039400000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":12.179400000000001,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":12.0221,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:23:49.8129008Z","DurationInMs":15.5713,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"eff3fcaa-c63d-4c9f-89ac-aeaf77c3b537","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.5721,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.5375,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:22:52.6615523Z","cpu":0.780,"memory":5251756.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2547,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:02.6764396Z","cpu":2.730,"memory":5355140.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1291,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:12.6863124Z","cpu":1.797,"memory":5348488.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0923,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:22.6975887Z","cpu":1.716,"memory":5343112.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1026,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:32.6992113Z","cpu":3.516,"memory":5337424.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1179,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:42.7033767Z","cpu":2.106,"memory":5326836.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1047,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.5307,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.5267,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.5049,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.5026,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.6632,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:23:49.8284721Z","EndTimeUTC":"2026-04-14T16:23:49.8440791Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:23:49.8598033Z","DurationInMs":15.724200000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"03ce06b4-e7b2-41e3-bebf-23a8d9c6eacb","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.716","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:23:49 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:23:49.8440791Z", "durationInMs": 0.0086},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:23:49.8440877Z", "durationInMs": 17.0212},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:23:49.8611089Z", "durationInMs": 0.1867},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:23:49.8612956Z", "durationInMs": 1.8463},{"event": "Received", "startTimeUtc": "2026-04-14T16:23:49.8631419Z", "durationInMs": 0.1249},{"event": "Completed", "startTimeUtc": "2026-04-14T16:23:49.8632668Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:23:49.8598033Z","lastSend":"2026-04-14T16:23:49.8598033Z","lastReceive":"2026-04-14T16:23:49.8598033Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1186,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:23:49.8284721Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"b321e168-72af-4395-9e38-053073bf9a6f","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:23:49.8284721Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"e058d523-6997-4ba4-88ac-fabbc622c6f8","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:23:49.8284721Z","DurationInMs":15.607000000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"03ce06b4-e7b2-41e3-bebf-23a8d9c6eacb","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1719773,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"03ce06b4-e7b2-41e3-bebf-23a8d9c6eacb","ResponseTimeUtc":"2026-04-14T16:23:49.8598033Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 03ce06b4-e7b2-41e3-bebf-23a8d9c6eacb, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.716"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0015}]}
2026-04-14 16:24:07 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 10046704-1eff-46a0-8358-ecb14b7443a7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 10046704-1eff-46a0-8358-ecb14b7443a7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 10046704-1eff-46a0-8358-ecb14b7443a7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 10046704-1eff-46a0-8358-ecb14b7443a7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:24:07.378Z","duration in milliseconds":56.554,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:24:07.2539310Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91069,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0477},{"name":"Get PkValue From Stream","duration in milliseconds":6.9986000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014},{"name":"Read Collection","duration in milliseconds":6.8474,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:24:07.3789371Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"ca45362c-24f5-46ad-88e7-2acefb026df4","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":49.286500000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":49.2504,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:23:12.6863124Z","cpu":1.797,"memory":5348488.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0923,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:22.6975887Z","cpu":1.716,"memory":5343112.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1026,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:32.6992113Z","cpu":3.516,"memory":5337424.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1179,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:42.7033767Z","cpu":2.106,"memory":5326836.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1047,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:52.7075932Z","cpu":1.641,"memory":5332604.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1945,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:24:02.7123919Z","cpu":2.656,"memory":5327052.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0841,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":49.2425,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":49.238,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":49.215700000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":49.211600000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":46.487300000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:24:07.4102565Z","EndTimeUTC":"2026-04-14T16:24:07.4102565Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:24:07.4258246Z","DurationInMs":15.568100000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"10046704-1eff-46a0-8358-ecb14b7443a7","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.9","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:24:07 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:24:07.4102565Z", "durationInMs": 0.0074},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:24:07.4102639Z", "durationInMs": 21.4574},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:24:07.4317213Z", "durationInMs": 0.2063},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:24:07.4319276Z", "durationInMs": 2.3918},{"event": "Received", "startTimeUtc": "2026-04-14T16:24:07.4343194Z", "durationInMs": 0.1462},{"event": "Completed", "startTimeUtc": "2026-04-14T16:24:07.4344656Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:24:07.4258246Z","lastSend":"2026-04-14T16:24:07.4258246Z","lastReceive":"2026-04-14T16:24:07.4258246Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1186,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:24:07.3945611Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"c3056fe1-2704-4142-8be7-68cc50849efd","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:24:07.3945611Z","DurationInMs":15.695400000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"83be2b50-d420-4760-a103-2be6ce709b74","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:24:07.4102565Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"10046704-1eff-46a0-8358-ecb14b7443a7","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.17189359999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"10046704-1eff-46a0-8358-ecb14b7443a7","ResponseTimeUtc":"2026-04-14T16:24:07.4414349Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 10046704-1eff-46a0-8358-ecb14b7443a7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.9"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 16:24:17 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: ab09169a-5d15-4e79-848d-e052648e94c6; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: ab09169a-5d15-4e79-848d-e052648e94c6, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: ab09169a-5d15-4e79-848d-e052648e94c6; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: ab09169a-5d15-4e79-848d-e052648e94c6, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:24:17.718Z","duration in milliseconds":19.531000000000002,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:24:17.3588556Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91075,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0456},{"name":"Get PkValue From Stream","duration in milliseconds":0.17420000000000002,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0013000000000000002}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":19.1405,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":19.1005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:23:22.6975887Z","cpu":1.716,"memory":5343112.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1026,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:32.6992113Z","cpu":3.516,"memory":5337424.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1179,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:42.7033767Z","cpu":2.106,"memory":5326836.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1047,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:52.7075932Z","cpu":1.641,"memory":5332604.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1945,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:24:02.7123919Z","cpu":2.656,"memory":5327052.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0841,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:24:12.7230290Z","cpu":4.290,"memory":5322404.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1011,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":19.0945,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":19.0902,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":19.0716,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":19.069000000000003,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":18.2234,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:24:17.7339202Z","DurationInMs":15.680900000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"ab09169a-5d15-4e79-848d-e052648e94c6","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.566","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:24:17 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:24:17.7182393Z", "durationInMs": 0.0153},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:24:17.7182546Z", "durationInMs": 15.0488},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:24:17.7333034Z", "durationInMs": 0.1863},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:24:17.7334897Z", "durationInMs": 2.0636},{"event": "Received", "startTimeUtc": "2026-04-14T16:24:17.7355533Z", "durationInMs": 0.123},{"event": "Completed", "startTimeUtc": "2026-04-14T16:24:17.7356763Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:24:17.7339202Z","lastSend":"2026-04-14T16:24:17.7339202Z","lastReceive":"2026-04-14T16:24:17.7339202Z"},"requestSizeInBytes":1631,"requestBodySizeInBytes":1183,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.37506459999999997,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"ab09169a-5d15-4e79-848d-e052648e94c6","ResponseTimeUtc":"2026-04-14T16:24:17.7339202Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: ab09169a-5d15-4e79-848d-e052648e94c6, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.566"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0013000000000000002}]}
2026-04-14 16:24:36 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6bc12407-fa0b-4a43-a908-4042a4566713; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6bc12407-fa0b-4a43-a908-4042a4566713, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6bc12407-fa0b-4a43-a908-4042a4566713; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6bc12407-fa0b-4a43-a908-4042a4566713, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:24:36.022Z","duration in milliseconds":44.5692,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:24:35.8667298Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91077,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0431},{"name":"Get PkValue From Stream","duration in milliseconds":6.2947000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":6.0992000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:24:36.0229852Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"343aa1fa-bf4e-4e17-acb3-bbbb2c16ffac","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":38.015100000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.9835,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:23:42.7033767Z","cpu":2.106,"memory":5326836.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1047,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:23:52.7075932Z","cpu":1.641,"memory":5332604.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1945,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:24:02.7123919Z","cpu":2.656,"memory":5327052.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0841,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:24:12.7230290Z","cpu":4.290,"memory":5322404.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1011,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:24:22.7334753Z","cpu":3.984,"memory":5328772.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0843,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:24:32.7381314Z","cpu":0.936,"memory":5321988.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0922,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.9744,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.97,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.9444,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.9406,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.4506,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:24:36.0386057Z","EndTimeUTC":"2026-04-14T16:24:36.0386057Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:24:36.0698561Z","DurationInMs":31.250400000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"6bc12407-fa0b-4a43-a908-4042a4566713","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.718","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:24:36 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:24:36.0386057Z", "durationInMs": 0.0091},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:24:36.0386148Z", "durationInMs": 19.646},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:24:36.0582608Z", "durationInMs": 0.212},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:24:36.0584728Z", "durationInMs": 2.0832},{"event": "Received", "startTimeUtc": "2026-04-14T16:24:36.0605560Z", "durationInMs": 0.201},{"event": "Completed", "startTimeUtc": "2026-04-14T16:24:36.0607570Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:24:36.0542328Z","lastSend":"2026-04-14T16:24:36.0542328Z","lastReceive":"2026-04-14T16:24:36.0542328Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1186,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:24:36.0229852Z","DurationInMs":15.620500000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"c6cf7a08-6e9a-4e13-8db9-9a813fb4886b","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:24:36.0386057Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"df22b084-212a-4f0e-a07a-c1a1546d538a","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:24:36.0386057Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"6bc12407-fa0b-4a43-a908-4042a4566713","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.20312629999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"6bc12407-fa0b-4a43-a908-4042a4566713","ResponseTimeUtc":"2026-04-14T16:24:36.0698561Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 6bc12407-fa0b-4a43-a908-4042a4566713, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.718"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0029000000000000002}]}
2026-04-14 16:25:01 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 36fdd012-8d1d-4c58-8e42-bfee44ae5f75; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 36fdd012-8d1d-4c58-8e42-bfee44ae5f75, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 36fdd012-8d1d-4c58-8e42-bfee44ae5f75; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 36fdd012-8d1d-4c58-8e42-bfee44ae5f75, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:25:01.587Z","duration in milliseconds":48.9031,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:25:01.4622467Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91087,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.044000000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":6.3041,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0013000000000000002},{"name":"Read Collection","duration in milliseconds":6.15,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:25:01.5874915Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"56c38f26-1cab-4977-82b0-4d99a1249c50","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":42.380900000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":42.347500000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:24:02.7123919Z","cpu":2.656,"memory":5327052.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0841,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:24:12.7230290Z","cpu":4.290,"memory":5322404.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1011,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:24:22.7334753Z","cpu":3.984,"memory":5328772.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0843,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:24:32.7381314Z","cpu":0.936,"memory":5321988.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0922,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:24:42.7446140Z","cpu":2.969,"memory":5311980.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3007,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:24:52.7478275Z","cpu":2.500,"memory":5312916.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2669,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":42.3404,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":42.3367,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":42.3175,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":42.314800000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":41.377700000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:25:01.6029271Z","EndTimeUTC":"2026-04-14T16:25:01.6029271Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:25:01.6342011Z","DurationInMs":31.274,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"36fdd012-8d1d-4c58-8e42-bfee44ae5f75","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"1.135","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:25:01 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:25:01.6029271Z", "durationInMs": 0.0087},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:25:01.6029358Z", "durationInMs": 23.9098},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:25:01.6268456Z", "durationInMs": 0.205},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:25:01.6270506Z", "durationInMs": 2.3138},{"event": "Received", "startTimeUtc": "2026-04-14T16:25:01.6293644Z", "durationInMs": 0.6416},{"event": "Completed", "startTimeUtc": "2026-04-14T16:25:01.6300060Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:25:01.6184988Z","lastSend":"2026-04-14T16:25:01.6184988Z","lastReceive":"2026-04-14T16:25:01.6184988Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:25:01.5874915Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"4f01ec8f-668a-4271-aa0a-db7935440cf3","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:25:01.5874915Z","DurationInMs":15.4356,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"d37d64f0-1de7-4ac7-8a8a-9ab4a30ddd4e","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:25:01.6029271Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"36fdd012-8d1d-4c58-8e42-bfee44ae5f75","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.17195439999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"36fdd012-8d1d-4c58-8e42-bfee44ae5f75","ResponseTimeUtc":"2026-04-14T16:25:01.6342011Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 36fdd012-8d1d-4c58-8e42-bfee44ae5f75, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"1.135"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 16:25:13 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: cc8db1f1-9043-40cd-8a42-dc06916d155f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: cc8db1f1-9043-40cd-8a42-dc06916d155f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: cc8db1f1-9043-40cd-8a42-dc06916d155f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: cc8db1f1-9043-40cd-8a42-dc06916d155f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:25:13.060Z","duration in milliseconds":43.6623,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:25:12.9668596Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91093,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0507},{"name":"Get PkValue From Stream","duration in milliseconds":7.4021000000000008,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0032},{"name":"Read Collection","duration in milliseconds":7.2288000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:25:13.0606156Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"6c9a1a3f-97af-4ff3-b6e1-fd4f89fd74c0","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.9686,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.9281,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:24:22.7334753Z","cpu":3.984,"memory":5328772.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0843,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:24:32.7381314Z","cpu":0.936,"memory":5321988.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0922,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:24:42.7446140Z","cpu":2.969,"memory":5311980.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3007,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:24:52.7478275Z","cpu":2.500,"memory":5312916.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2669,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:25:02.7642867Z","cpu":3.666,"memory":5293428.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.064,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:25:12.7793745Z","cpu":2.106,"memory":5291216.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0568,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.9196,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.914500000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.8888,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.8832,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.2847,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:25:13.0763230Z","EndTimeUTC":"2026-04-14T16:25:13.0763230Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:25:13.1074875Z","DurationInMs":31.1645,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"cc8db1f1-9043-40cd-8a42-dc06916d155f","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.535","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:25:13 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:25:13.0763230Z", "durationInMs": 0.0099},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:25:13.0763329Z", "durationInMs": 15.8816},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:25:13.0922145Z", "durationInMs": 0.1781},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:25:13.0923926Z", "durationInMs": 1.7985},{"event": "Received", "startTimeUtc": "2026-04-14T16:25:13.0941911Z", "durationInMs": 0.1243},{"event": "Completed", "startTimeUtc": "2026-04-14T16:25:13.0943154Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:25:13.0918640Z","lastSend":"2026-04-14T16:25:13.0918640Z","lastReceive":"2026-04-14T16:25:13.0918640Z"},"requestSizeInBytes":1630,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:25:13.0606156Z","DurationInMs":15.707400000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"2328307d-01af-445e-972e-44e4d6a145f6","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:25:13.0763230Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"e3971b70-666d-43c7-b707-2ab76d6aa710","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:25:13.0763230Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"cc8db1f1-9043-40cd-8a42-dc06916d155f","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1406279,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"cc8db1f1-9043-40cd-8a42-dc06916d155f","ResponseTimeUtc":"2026-04-14T16:25:13.1074875Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: cc8db1f1-9043-40cd-8a42-dc06916d155f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.535"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0027}]}
2026-04-14 16:27:36 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e08faa9b-8b6b-4a34-ad09-a0e812fd66fe; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e08faa9b-8b6b-4a34-ad09-a0e812fd66fe, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e08faa9b-8b6b-4a34-ad09-a0e812fd66fe; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e08faa9b-8b6b-4a34-ad09-a0e812fd66fe, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:27:36.799Z","duration in milliseconds":20.1008,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:27:36.4246934Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91155,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0512},{"name":"Get PkValue From Stream","duration in milliseconds":0.1565,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0026000000000000003}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":19.7212,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":19.688100000000002,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:26:42.8518677Z","cpu":2.031,"memory":5261084.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1309,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:26:52.8536426Z","cpu":2.500,"memory":5258904.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1753,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:27:02.8623714Z","cpu":4.453,"memory":5257800.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.102,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:27:12.8727749Z","cpu":5.226,"memory":5257596.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.265,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:27:22.8800513Z","cpu":5.538,"memory":5258528.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1348,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:27:32.8918818Z","cpu":3.281,"memory":5217632.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.164,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":19.6827,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":19.6785,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":19.6598,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":19.657700000000002,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":18.6736,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:27:36.8153257Z","DurationInMs":15.502600000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"e08faa9b-8b6b-4a34-ad09-a0e812fd66fe","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.647","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:27:36 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:27:36.7998231Z", "durationInMs": 0.013},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:27:36.7998361Z", "durationInMs": 15.4638},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:27:36.8152999Z", "durationInMs": 0.1887},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:27:36.8154886Z", "durationInMs": 2.0969},{"event": "Received", "startTimeUtc": "2026-04-14T16:27:36.8175855Z", "durationInMs": 0.1365},{"event": "Completed", "startTimeUtc": "2026-04-14T16:27:36.8177220Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:27:36.7998231Z","lastSend":"2026-04-14T16:27:36.7998231Z","lastReceive":"2026-04-14T16:27:36.8153257Z"},"requestSizeInBytes":1612,"requestBodySizeInBytes":1164,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.3906323,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"e08faa9b-8b6b-4a34-ad09-a0e812fd66fe","ResponseTimeUtc":"2026-04-14T16:27:36.8153257Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: e08faa9b-8b6b-4a34-ad09-a0e812fd66fe, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.647"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001}]}
2026-04-14 16:33:51 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b369f56e-0b35-43f9-892f-f8e4ce75063e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b369f56e-0b35-43f9-892f-f8e4ce75063e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b369f56e-0b35-43f9-892f-f8e4ce75063e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b369f56e-0b35-43f9-892f-f8e4ce75063e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:33:51.600Z","duration in milliseconds":23.7334,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:33:51.2255448Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91266,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0531},{"name":"Get PkValue From Stream","duration in milliseconds":0.1648,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":23.331300000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":23.297800000000002,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:32:53.2122909Z","cpu":4.922,"memory":5238904.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2568,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:33:03.2243886Z","cpu":4.836,"memory":5291708.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1319,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:33:13.2373916Z","cpu":2.184,"memory":5283944.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0828,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:33:23.2436335Z","cpu":1.250,"memory":5278148.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0983,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:33:33.2572658Z","cpu":5.772,"memory":5275940.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0962,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:33:43.2620901Z","cpu":8.346,"memory":5263972.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0518,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":23.2907,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":23.2863,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":23.2645,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":23.262,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":22.1491,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:33:51.6161801Z","DurationInMs":15.627600000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"b369f56e-0b35-43f9-892f-f8e4ce75063e","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149388,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149388,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149388,"SessionToken":"-1#26149388","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.643","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:33:51 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:33:51.6005525Z", "durationInMs": 0.0125},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:33:51.6005650Z", "durationInMs": 18.8953},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:33:51.6194603Z", "durationInMs": 0.1917},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:33:51.6196520Z", "durationInMs": 2.1358},{"event": "Received", "startTimeUtc": "2026-04-14T16:33:51.6217878Z", "durationInMs": 0.1357},{"event": "Completed", "startTimeUtc": "2026-04-14T16:33:51.6219235Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:33:51.6161801Z","lastSend":"2026-04-14T16:33:51.6161801Z","lastReceive":"2026-04-14T16:33:51.6161801Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1186,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.39063529999999996,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"b369f56e-0b35-43f9-892f-f8e4ce75063e","ResponseTimeUtc":"2026-04-14T16:33:51.6161801Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: b369f56e-0b35-43f9-892f-f8e4ce75063e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149388","BELatencyInMs":"0.643"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 16:36:57 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 9e530c26-8c0c-4a3c-b888-55d6f1bf2698; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 9e530c26-8c0c-4a3c-b888-55d6f1bf2698, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 9e530c26-8c0c-4a3c-b888-55d6f1bf2698; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 9e530c26-8c0c-4a3c-b888-55d6f1bf2698, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:36:57.095Z","duration in milliseconds":58.814600000000006,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:36:56.6112600Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91297,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0414},{"name":"Get PkValue From Stream","duration in milliseconds":6.3902,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.2426,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:36:57.0956486Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"32701094-e191-488f-98b5-42ef66d6202b","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":52.208400000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":52.1758,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:36:03.3732420Z","cpu":2.730,"memory":5201684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.087,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:36:13.3816360Z","cpu":1.250,"memory":5202156.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1057,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:36:23.3875781Z","cpu":3.203,"memory":5225496.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1052,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:36:33.4006961Z","cpu":0.468,"memory":5217876.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2207,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:36:43.4043119Z","cpu":0.859,"memory":5217528.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1327,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:36:53.4138788Z","cpu":2.262,"memory":5306268.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1088,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":52.1681,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":52.164,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":52.145,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":52.142500000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":51.1695,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:36:57.1113378Z","EndTimeUTC":"2026-04-14T16:36:57.1113378Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:36:57.1581473Z","DurationInMs":46.8095,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"9e530c26-8c0c-4a3c-b888-55d6f1bf2698","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.647","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:36:57 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:36:57.1113378Z", "durationInMs": 0.0079},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:36:57.1113457Z", "durationInMs": 33.3591},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:36:57.1447048Z", "durationInMs": 0.1995},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:36:57.1449043Z", "durationInMs": 1.9348},{"event": "Received", "startTimeUtc": "2026-04-14T16:36:57.1468391Z", "durationInMs": 0.1367},{"event": "Completed", "startTimeUtc": "2026-04-14T16:36:57.1469758Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:36:57.1425231Z","lastSend":"2026-04-14T16:36:57.1425231Z","lastReceive":"2026-04-14T16:36:57.1425231Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:36:57.0956486Z","DurationInMs":15.689200000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"50371ffd-02c9-4ed5-be4d-cc4ed1669fcb","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:36:57.1113378Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"3e6f689a-490f-4cb6-864d-54ff12640f44","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:36:57.1113378Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"9e530c26-8c0c-4a3c-b888-55d6f1bf2698","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.5468873,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"9e530c26-8c0c-4a3c-b888-55d6f1bf2698","ResponseTimeUtc":"2026-04-14T16:36:57.1581473Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 9e530c26-8c0c-4a3c-b888-55d6f1bf2698, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.647"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0015}]}
2026-04-14 16:50:33 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5721e377-8965-4d5a-9856-39918edf367f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5721e377-8965-4d5a-9856-39918edf367f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5721e377-8965-4d5a-9856-39918edf367f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5721e377-8965-4d5a-9856-39918edf367f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:50:33.297Z","duration in milliseconds":43.5109,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:50:33.1570031Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91323,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0408},{"name":"Get PkValue From Stream","duration in milliseconds":7.162,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.00090000000000000008},{"name":"Read Collection","duration in milliseconds":7.0105,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:50:33.2977325Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"1616ef74-cd35-464c-9419-e144d9b78e7d","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.111000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.0745,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:49:34.0825928Z","cpu":1.170,"memory":5286768.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2667,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:49:44.0850398Z","cpu":0.703,"memory":5286340.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1706,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:49:54.1003646Z","cpu":0.780,"memory":5279816.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1307,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:50:04.1013627Z","cpu":1.875,"memory":5263036.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2464,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:50:14.1114502Z","cpu":4.524,"memory":5162892.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1772,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:50:24.1200138Z","cpu":3.203,"memory":5164592.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.238,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.062200000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.057,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.0311,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.0282,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.9974,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:50:33.3132708Z","EndTimeUTC":"2026-04-14T16:50:33.3132708Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:50:33.3288820Z","DurationInMs":15.6112,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"5721e377-8965-4d5a-9856-39918edf367f","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.827","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:50:33 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:50:33.3132708Z", "durationInMs": 0.0086},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:50:33.3132794Z", "durationInMs": 17.9489},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:50:33.3312283Z", "durationInMs": 0.2017},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:50:33.3314300Z", "durationInMs": 2.1713},{"event": "Received", "startTimeUtc": "2026-04-14T16:50:33.3336013Z", "durationInMs": 0.1311},{"event": "Completed", "startTimeUtc": "2026-04-14T16:50:33.3337324Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:50:33.3288820Z","lastSend":"2026-04-14T16:50:33.3288820Z","lastReceive":"2026-04-14T16:50:33.3288820Z"},"requestSizeInBytes":1641,"requestBodySizeInBytes":1187,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:50:33.2977325Z","DurationInMs":15.538300000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"4a938bd7-af23-4329-9d42-4f2923d2bb16","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:50:33.3132708Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"dfad4d4d-e914-4298-8a10-2ec2d361913f","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:50:33.3132708Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"5721e377-8965-4d5a-9856-39918edf367f","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718789,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"5721e377-8965-4d5a-9856-39918edf367f","ResponseTimeUtc":"2026-04-14T16:50:33.3288820Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 5721e377-8965-4d5a-9856-39918edf367f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.827"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.003}]}
2026-04-14 16:50:54 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6a8052b1-8bc6-4b40-b0af-2a4738c17194; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6a8052b1-8bc6-4b40-b0af-2a4738c17194, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6a8052b1-8bc6-4b40-b0af-2a4738c17194; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6a8052b1-8bc6-4b40-b0af-2a4738c17194, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:50:54.075Z","duration in milliseconds":43.993300000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:50:53.9500118Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91328,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0531},{"name":"Get PkValue From Stream","duration in milliseconds":6.0994,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0044},{"name":"Read Collection","duration in milliseconds":5.9238,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:50:54.0750129Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"ffe95d5f-151b-4589-9a63-4827bc8c7845","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.6587,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.6212,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:49:54.1003646Z","cpu":0.780,"memory":5279816.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1307,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:50:04.1013627Z","cpu":1.875,"memory":5263036.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2464,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:50:14.1114502Z","cpu":4.524,"memory":5162892.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1772,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:50:24.1200138Z","cpu":3.203,"memory":5164592.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.238,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:50:34.1257706Z","cpu":0.936,"memory":5192692.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2454,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:50:44.1330580Z","cpu":4.922,"memory":5137412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.067,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.613600000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.6089,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.587900000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.5857,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.6878,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:50:54.0906384Z","EndTimeUTC":"2026-04-14T16:50:54.0906384Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:50:54.1218905Z","DurationInMs":31.252100000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"6a8052b1-8bc6-4b40-b0af-2a4738c17194","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.58","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:50:54 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:50:54.0906384Z", "durationInMs": 0.0086},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:50:54.0906470Z", "durationInMs": 18.1471},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:50:54.1087941Z", "durationInMs": 0.1885},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:50:54.1089826Z", "durationInMs": 2.0192},{"event": "Received", "startTimeUtc": "2026-04-14T16:50:54.1110018Z", "durationInMs": 0.135},{"event": "Completed", "startTimeUtc": "2026-04-14T16:50:54.1111368Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:50:54.1062641Z","lastSend":"2026-04-14T16:50:54.1062641Z","lastReceive":"2026-04-14T16:50:54.1218905Z"},"requestSizeInBytes":1635,"requestBodySizeInBytes":1187,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:50:54.0750129Z","DurationInMs":15.6255,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"bf0dadff-51a9-4324-b5ed-ea859582ae0e","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:50:54.0906384Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"93fa1069-ff80-4723-a159-46cf06e377e9","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:50:54.0906384Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"6a8052b1-8bc6-4b40-b0af-2a4738c17194","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718787,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"6a8052b1-8bc6-4b40-b0af-2a4738c17194","ResponseTimeUtc":"2026-04-14T16:50:54.1218905Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 6a8052b1-8bc6-4b40-b0af-2a4738c17194, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.58"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 16:51:22 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c0ae6994-3bdb-4c8c-aee7-a296c5ee8527; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c0ae6994-3bdb-4c8c-aee7-a296c5ee8527, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c0ae6994-3bdb-4c8c-aee7-a296c5ee8527; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c0ae6994-3bdb-4c8c-aee7-a296c5ee8527, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:51:22.337Z","duration in milliseconds":46.8305,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:51:22.2276427Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91331,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0495},{"name":"Get PkValue From Stream","duration in milliseconds":7.8998000000000008,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0039000000000000003},{"name":"Read Collection","duration in milliseconds":7.7029000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:51:22.3370235Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"297b5ffc-0ab3-4b70-9016-88062805a191","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":38.688700000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":38.6473,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:50:24.1200138Z","cpu":3.203,"memory":5164592.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.238,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:50:34.1257706Z","cpu":0.936,"memory":5192692.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2454,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:50:44.1330580Z","cpu":4.922,"memory":5137412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.067,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:50:54.1375179Z","cpu":1.094,"memory":5135660.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0907,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:51:04.1414506Z","cpu":1.950,"memory":5131252.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0356,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:51:14.1553215Z","cpu":0.469,"memory":5131284.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1283,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":38.6388,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":38.633700000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":38.6154,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":38.612500000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":37.6178,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:51:22.3526452Z","EndTimeUTC":"2026-04-14T16:51:22.3682663Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:51:22.3838979Z","DurationInMs":15.6316,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c0ae6994-3bdb-4c8c-aee7-a296c5ee8527","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.593","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:51:22 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:51:22.3682663Z", "durationInMs": 0.0098},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:51:22.3682761Z", "durationInMs": 19.0855},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:51:22.3873616Z", "durationInMs": 0.2175},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:51:22.3875791Z", "durationInMs": 1.7918},{"event": "Received", "startTimeUtc": "2026-04-14T16:51:22.3893709Z", "durationInMs": 0.1368},{"event": "Completed", "startTimeUtc": "2026-04-14T16:51:22.3895077Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:51:22.3838979Z","lastSend":"2026-04-14T16:51:22.3838979Z","lastReceive":"2026-04-14T16:51:22.3838979Z"},"requestSizeInBytes":1639,"requestBodySizeInBytes":1187,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:51:22.3526452Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"c5984858-c54d-4e0a-9927-a3ee27fe6051","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:51:22.3526452Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0001b9d0-7f6d-4c72-a88b-fe3eb115e676","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:51:22.3526452Z","DurationInMs":15.6211,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"c0ae6994-3bdb-4c8c-aee7-a296c5ee8527","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.15625519999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c0ae6994-3bdb-4c8c-aee7-a296c5ee8527","ResponseTimeUtc":"2026-04-14T16:51:22.3838979Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c0ae6994-3bdb-4c8c-aee7-a296c5ee8527, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.593"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 16:51:39 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6f022ae0-a991-4fc4-a2a4-f2068a7248c4; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6f022ae0-a991-4fc4-a2a4-f2068a7248c4, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6f022ae0-a991-4fc4-a2a4-f2068a7248c4; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6f022ae0-a991-4fc4-a2a4-f2068a7248c4, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:51:39.607Z","duration in milliseconds":58.0308,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:51:39.4980096Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91335,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0403},{"name":"Get PkValue From Stream","duration in milliseconds":7.9328,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":7.7803,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:51:39.6073869Z","DurationInMs":15.688500000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"762abd44-624c-4b21-a777-ba5f4a7581f8","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":49.8813,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":49.851,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:50:44.1330580Z","cpu":4.922,"memory":5137412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.067,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:50:54.1375179Z","cpu":1.094,"memory":5135660.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0907,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:51:04.1414506Z","cpu":1.950,"memory":5131252.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0356,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:51:14.1553215Z","cpu":0.469,"memory":5131284.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1283,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:51:24.1579664Z","cpu":2.496,"memory":5132280.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0726,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:51:34.1742975Z","cpu":2.496,"memory":5134684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1226,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":49.8414,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":49.832,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":49.8127,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":49.810700000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":48.967200000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:51:39.6230754Z","EndTimeUTC":"2026-04-14T16:51:39.6386391Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:51:39.6698863Z","DurationInMs":31.247200000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"6f022ae0-a991-4fc4-a2a4-f2068a7248c4","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.597","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:51:39 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:51:39.6386391Z", "durationInMs": 0.0087},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:51:39.6386478Z", "durationInMs": 31.9699},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:51:39.6706177Z", "durationInMs": 0.2081},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:51:39.6708258Z", "durationInMs": 1.9406},{"event": "Received", "startTimeUtc": "2026-04-14T16:51:39.6727664Z", "durationInMs": 0.1384},{"event": "Completed", "startTimeUtc": "2026-04-14T16:51:39.6729048Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:51:39.6698863Z","lastSend":"2026-04-14T16:51:39.6698863Z","lastReceive":"2026-04-14T16:51:39.6698863Z"},"requestSizeInBytes":1637,"requestBodySizeInBytes":1189,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:51:39.6230754Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"87675f0a-c372-4d58-ab7e-3c4a6c581287","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:51:39.6230754Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"09ff6c14-4f35-42e4-b6e6-5b8791e63a4a","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:51:39.6230754Z","DurationInMs":15.5637,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"6f022ae0-a991-4fc4-a2a4-f2068a7248c4","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1875025,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"6f022ae0-a991-4fc4-a2a4-f2068a7248c4","ResponseTimeUtc":"2026-04-14T16:51:39.6698863Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 6f022ae0-a991-4fc4-a2a4-f2068a7248c4, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.597"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0026000000000000003}]}
2026-04-14 16:52:05 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 702b9453-65ea-4f31-9c8a-c65e434e3aa7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 702b9453-65ea-4f31-9c8a-c65e434e3aa7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 702b9453-65ea-4f31-9c8a-c65e434e3aa7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 702b9453-65ea-4f31-9c8a-c65e434e3aa7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:52:05.675Z","duration in milliseconds":40.7533,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:52:05.5193862Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91339,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0412},{"name":"Get PkValue From Stream","duration in milliseconds":5.8475,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0051},{"name":"Read Collection","duration in milliseconds":5.7073,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:52:05.6756892Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"dadb7bd1-c484-4117-aa31-3300a540d2e5","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":34.7053,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":34.6799,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:51:14.1553215Z","cpu":0.469,"memory":5131284.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1283,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:51:24.1579664Z","cpu":2.496,"memory":5132280.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0726,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:51:34.1742975Z","cpu":2.496,"memory":5134684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1226,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:51:44.1833491Z","cpu":1.797,"memory":5135380.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0764,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:51:54.1902747Z","cpu":2.262,"memory":5137448.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3557,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:52:04.1912572Z","cpu":5.859,"memory":5127704.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.074,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":34.673100000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":34.6693,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":34.6484,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":34.646300000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":33.8269,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:52:05.6912636Z","EndTimeUTC":"2026-04-14T16:52:05.6912636Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:52:05.7068987Z","DurationInMs":15.635100000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"702b9453-65ea-4f31-9c8a-c65e434e3aa7","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.708","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:52:05 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:52:05.6912636Z", "durationInMs": 0.0078},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:52:05.6912714Z", "durationInMs": 17.247},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:52:05.7085184Z", "durationInMs": 0.1918},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:52:05.7087102Z", "durationInMs": 2.0499},{"event": "Received", "startTimeUtc": "2026-04-14T16:52:05.7107601Z", "durationInMs": 0.1165},{"event": "Completed", "startTimeUtc": "2026-04-14T16:52:05.7108766Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:52:05.7068987Z","lastSend":"2026-04-14T16:52:05.7068987Z","lastReceive":"2026-04-14T16:52:05.7068987Z"},"requestSizeInBytes":1641,"requestBodySizeInBytes":1189,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:52:05.6756892Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"27e4bb3c-a95f-4a0e-955d-c4b37fd1d2d7","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:52:05.6756892Z","DurationInMs":15.5744,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"73e650d5-9f40-4b86-b6cb-626d74f0d29e","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:52:05.6912636Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"702b9453-65ea-4f31-9c8a-c65e434e3aa7","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2031376,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"702b9453-65ea-4f31-9c8a-c65e434e3aa7","ResponseTimeUtc":"2026-04-14T16:52:05.7068987Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 702b9453-65ea-4f31-9c8a-c65e434e3aa7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.708"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0014}]}
2026-04-14 16:52:31 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6b1ef4ca-1771-4253-8587-ad24a6c11278; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6b1ef4ca-1771-4253-8587-ad24a6c11278, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6b1ef4ca-1771-4253-8587-ad24a6c11278; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6b1ef4ca-1771-4253-8587-ad24a6c11278, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:52:31.937Z","duration in milliseconds":44.9974,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:52:31.8122920Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91341,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.046},{"name":"Get PkValue From Stream","duration in milliseconds":6.6510000000000007,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014},{"name":"Read Collection","duration in milliseconds":6.5020000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:52:31.9372955Z","DurationInMs":15.6902,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"e8ed36b8-0dd2-4c07-9f25-74c29e04574b","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":38.118900000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":38.087900000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:51:34.1742975Z","cpu":2.496,"memory":5134684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1226,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:51:44.1833491Z","cpu":1.797,"memory":5135380.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0764,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:51:54.1902747Z","cpu":2.262,"memory":5137448.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3557,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:52:04.1912572Z","cpu":5.859,"memory":5127704.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.074,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:52:14.1957592Z","cpu":3.047,"memory":5125180.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1553,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:52:24.2114074Z","cpu":0.702,"memory":5141108.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0345,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":38.0811,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":38.077000000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":38.0561,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":38.054,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":37.149300000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:52:31.9529857Z","EndTimeUTC":"2026-04-14T16:52:31.9685476Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:52:31.9841695Z","DurationInMs":15.6219,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"6b1ef4ca-1771-4253-8587-ad24a6c11278","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.647","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:52:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:52:31.9685476Z", "durationInMs": 0.009},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:52:31.9685566Z", "durationInMs": 17.8098},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:52:31.9863664Z", "durationInMs": 0.1936},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:52:31.9865600Z", "durationInMs": 4.1323},{"event": "Received", "startTimeUtc": "2026-04-14T16:52:31.9906923Z", "durationInMs": 0.1424},{"event": "Completed", "startTimeUtc": "2026-04-14T16:52:31.9908347Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:52:31.9841695Z","lastSend":"2026-04-14T16:52:31.9841695Z","lastReceive":"2026-04-14T16:52:31.9841695Z"},"requestSizeInBytes":1637,"requestBodySizeInBytes":1187,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:52:31.9529857Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"ebedd114-f3dc-43f8-ae20-2856ab021041","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:52:31.9529857Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"19ad4508-baba-42ea-bd1f-b3374a6f5ad0","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:52:31.9529857Z","DurationInMs":15.561900000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"6b1ef4ca-1771-4253-8587-ad24a6c11278","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718775,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"6b1ef4ca-1771-4253-8587-ad24a6c11278","ResponseTimeUtc":"2026-04-14T16:52:31.9841695Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 6b1ef4ca-1771-4253-8587-ad24a6c11278, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.647"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 16:52:53 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 4faa093b-e1f2-424a-8432-5ade91761c8f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 4faa093b-e1f2-424a-8432-5ade91761c8f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 4faa093b-e1f2-424a-8432-5ade91761c8f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 4faa093b-e1f2-424a-8432-5ade91761c8f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:52:53.925Z","duration in milliseconds":50.697700000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:52:53.7849063Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91346,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0429},{"name":"Get PkValue From Stream","duration in milliseconds":6.7791000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014},{"name":"Read Collection","duration in milliseconds":6.6253,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:52:53.9255965Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"0c4a39bd-a137-4a34-ab44-acf0678d1a1f","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":43.7048,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":43.671800000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:51:54.1902747Z","cpu":2.262,"memory":5137448.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3557,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:52:04.1912572Z","cpu":5.859,"memory":5127704.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.074,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:52:14.1957592Z","cpu":3.047,"memory":5125180.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1553,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:52:24.2114074Z","cpu":0.702,"memory":5141108.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0345,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:52:34.2231014Z","cpu":1.482,"memory":5141128.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0773,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:52:44.2398306Z","cpu":2.418,"memory":5140468.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.136,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":43.664500000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":43.660000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":43.6417,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":43.639500000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":42.666900000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:52:53.9412256Z","EndTimeUTC":"2026-04-14T16:52:53.9412256Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:52:53.9724070Z","DurationInMs":31.1814,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"4faa093b-e1f2-424a-8432-5ade91761c8f","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.569","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:52:53 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:52:53.9412256Z", "durationInMs": 0.0094},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:52:53.9412350Z", "durationInMs": 25.9684},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:52:53.9672034Z", "durationInMs": 0.1958},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:52:53.9673992Z", "durationInMs": 1.9687},{"event": "Received", "startTimeUtc": "2026-04-14T16:52:53.9693679Z", "durationInMs": 0.1425},{"event": "Completed", "startTimeUtc": "2026-04-14T16:52:53.9695104Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:52:53.9567801Z","lastSend":"2026-04-14T16:52:53.9567801Z","lastReceive":"2026-04-14T16:52:53.9724070Z"},"requestSizeInBytes":1635,"requestBodySizeInBytes":1187,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:52:53.9255965Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"66d006a5-194a-4ee8-9393-c29cbc969055","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:52:53.9255965Z","DurationInMs":15.629100000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"2098f96e-38e9-4629-9bb2-ca944a39b620","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:52:53.9412256Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"4faa093b-e1f2-424a-8432-5ade91761c8f","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2031299,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"4faa093b-e1f2-424a-8432-5ade91761c8f","ResponseTimeUtc":"2026-04-14T16:52:53.9724070Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 4faa093b-e1f2-424a-8432-5ade91761c8f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.569"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 16:53:49 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 32399ecd-c99a-41b4-acaf-96f411967ad6; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 32399ecd-c99a-41b4-acaf-96f411967ad6, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 32399ecd-c99a-41b4-acaf-96f411967ad6; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 32399ecd-c99a-41b4-acaf-96f411967ad6, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:53:49.580Z","duration in milliseconds":43.1486,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:53:49.4867781Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91350,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0505},{"name":"Get PkValue From Stream","duration in milliseconds":6.1494,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0015},{"name":"Read Collection","duration in milliseconds":5.9931,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:53:49.5805311Z","DurationInMs":15.6783,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"31c3f958-0dca-48d7-83a6-a46675dc6363","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.7654,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.732,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:52:54.2536613Z","cpu":2.886,"memory":5211432.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1132,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:53:04.2688480Z","cpu":0.858,"memory":5362760.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0775,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:53:14.2760494Z","cpu":4.688,"memory":5357296.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1226,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:53:24.2834285Z","cpu":6.094,"memory":5352472.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2663,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:53:34.2963011Z","cpu":4.368,"memory":5348600.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0918,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:53:44.3123531Z","cpu":4.758,"memory":5356192.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0897,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.7244,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.7202,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.7021,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.6982,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.8176,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:53:49.5962094Z","EndTimeUTC":"2026-04-14T16:53:49.5962094Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:53:49.6274046Z","DurationInMs":31.1952,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"32399ecd-c99a-41b4-acaf-96f411967ad6","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.688","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:53:49 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:53:49.5962094Z", "durationInMs": 0.0089},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:53:49.5962183Z", "durationInMs": 19.1363},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:53:49.6153546Z", "durationInMs": 0.163},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:53:49.6155176Z", "durationInMs": 2.4132},{"event": "Received", "startTimeUtc": "2026-04-14T16:53:49.6179308Z", "durationInMs": 0.1113},{"event": "Completed", "startTimeUtc": "2026-04-14T16:53:49.6180421Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:53:49.6274046Z","lastSend":"2026-04-14T16:53:49.6274046Z","lastReceive":"2026-04-14T16:53:49.6274046Z"},"requestSizeInBytes":1635,"requestBodySizeInBytes":1187,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:53:49.5962094Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a03d3754-9e62-4136-9732-89e64f246ecb","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:53:49.5962094Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0e70265c-d418-480f-8f5f-d315654e4dcf","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:53:49.5962094Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"32399ecd-c99a-41b4-acaf-96f411967ad6","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1406265,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"32399ecd-c99a-41b4-acaf-96f411967ad6","ResponseTimeUtc":"2026-04-14T16:53:49.6274046Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 32399ecd-c99a-41b4-acaf-96f411967ad6, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.688"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 16:53:57 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 45fd9ee8-3434-450d-aded-c5fde8a96a30; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 45fd9ee8-3434-450d-aded-c5fde8a96a30, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 45fd9ee8-3434-450d-aded-c5fde8a96a30; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 45fd9ee8-3434-450d-aded-c5fde8a96a30, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:53:57.167Z","duration in milliseconds":45.6858,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:53:57.0896673Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91354,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.045700000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":8.7757,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0019},{"name":"Read Collection","duration in milliseconds":8.5084,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:53:57.1677961Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"c81e2e18-0ffb-4bd7-ac3c-119bbc2fb736","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.6719,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.6407,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:53:04.2688480Z","cpu":0.858,"memory":5362760.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0775,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:53:14.2760494Z","cpu":4.688,"memory":5357296.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1226,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:53:24.2834285Z","cpu":6.094,"memory":5352472.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2663,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:53:34.2963011Z","cpu":4.368,"memory":5348600.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0918,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:53:44.3123531Z","cpu":4.758,"memory":5356192.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0897,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:53:54.3150719Z","cpu":9.844,"memory":5340540.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1164,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.6327,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.6293,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.609100000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.6059,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.3635,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:53:57.1834179Z","EndTimeUTC":"2026-04-14T16:53:57.1834179Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:53:57.1990419Z","DurationInMs":15.624,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"45fd9ee8-3434-450d-aded-c5fde8a96a30","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.61","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:53:57 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:53:57.1834179Z", "durationInMs": 0.0054},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:53:57.1834233Z", "durationInMs": 18.7744},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:53:57.2021977Z", "durationInMs": 0.1782},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:53:57.2023759Z", "durationInMs": 2.1801},{"event": "Received", "startTimeUtc": "2026-04-14T16:53:57.2045560Z", "durationInMs": 0.1254},{"event": "Completed", "startTimeUtc": "2026-04-14T16:53:57.2046814Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:53:57.1990419Z","lastSend":"2026-04-14T16:53:57.1990419Z","lastReceive":"2026-04-14T16:53:57.1990419Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:53:57.1677961Z","DurationInMs":15.6218,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"3b035bda-582b-4b00-9193-59235de1bc0d","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:53:57.1834179Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"46c025c3-e484-46af-a423-60bc82a344a4","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:53:57.1834179Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"45fd9ee8-3434-450d-aded-c5fde8a96a30","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.10937459999999999,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"45fd9ee8-3434-450d-aded-c5fde8a96a30","ResponseTimeUtc":"2026-04-14T16:53:57.2146674Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 45fd9ee8-3434-450d-aded-c5fde8a96a30, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.61"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 16:55:47 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 921fd2e9-6c88-451d-8395-1ac3fcae34db; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 921fd2e9-6c88-451d-8395-1ac3fcae34db, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 921fd2e9-6c88-451d-8395-1ac3fcae34db; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 921fd2e9-6c88-451d-8395-1ac3fcae34db, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:55:47.552Z","duration in milliseconds":73.161,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:55:47.3961513Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91366,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0407},{"name":"Get PkValue From Stream","duration in milliseconds":9.2134,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.001},{"name":"Read Collection","duration in milliseconds":9.0522000000000009,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:55:47.5524577Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"12ecc177-6aca-4c87-8f4d-86666dafe28a","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":63.7316,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":63.698600000000006,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:54:54.3642717Z","cpu":2.262,"memory":5312096.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1265,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:55:04.3697253Z","cpu":2.188,"memory":5294804.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1251,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:55:14.3759811Z","cpu":2.188,"memory":5291268.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0885,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:55:24.3903703Z","cpu":1.872,"memory":5293576.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1513,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:55:34.4053422Z","cpu":0.390,"memory":5292600.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0723,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:55:44.4117704Z","cpu":3.042,"memory":5293172.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2134,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":63.6916,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":63.6873,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":63.6664,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":63.6634,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":62.698100000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:55:47.5682073Z","EndTimeUTC":"2026-04-14T16:55:47.5682073Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:55:47.6149066Z","DurationInMs":46.6993,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"921fd2e9-6c88-451d-8395-1ac3fcae34db","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.711","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:55:47 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:55:47.5682073Z", "durationInMs": 0.0088},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:55:47.5682161Z", "durationInMs": 44.1869},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:55:47.6124030Z", "durationInMs": 0.1995},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:55:47.6126025Z", "durationInMs": 1.8206},{"event": "Received", "startTimeUtc": "2026-04-14T16:55:47.6144231Z", "durationInMs": 0.1503},{"event": "Completed", "startTimeUtc": "2026-04-14T16:55:47.6145734Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:55:47.6149066Z","lastSend":"2026-04-14T16:55:47.6149066Z","lastReceive":"2026-04-14T16:55:47.6149066Z"},"requestSizeInBytes":1637,"requestBodySizeInBytes":1187,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:55:47.5524577Z","DurationInMs":15.749600000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"61b84cf2-a792-4e83-a97f-8d045d021801","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:55:47.5682073Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"ba112324-2ba7-407d-bc23-06488ed81310","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:55:47.5682073Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"921fd2e9-6c88-451d-8395-1ac3fcae34db","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.21875529999999999,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"921fd2e9-6c88-451d-8395-1ac3fcae34db","ResponseTimeUtc":"2026-04-14T16:55:47.6149066Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 921fd2e9-6c88-451d-8395-1ac3fcae34db, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.711"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 16:58:37 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: dcbc993b-13d2-47fd-aa8f-2436509f499f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: dcbc993b-13d2-47fd-aa8f-2436509f499f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: dcbc993b-13d2-47fd-aa8f-2436509f499f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: dcbc993b-13d2-47fd-aa8f-2436509f499f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T16:58:37.918Z","duration in milliseconds":46.4193,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T16:58:37.5275945Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91383,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0427},{"name":"Get PkValue From Stream","duration in milliseconds":6.5932,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0015},{"name":"Read Collection","duration in milliseconds":6.4556000000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:58:37.9182936Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"06314154-6679-40f8-a972-4b50b2aae39a","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":39.6088,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":39.579100000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T16:57:44.5418228Z","cpu":2.730,"memory":5255840.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0795,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:57:54.5525415Z","cpu":1.641,"memory":5260604.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2494,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:58:04.5616644Z","cpu":1.560,"memory":5314384.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0902,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:58:14.5669564Z","cpu":1.484,"memory":5309108.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2774,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:58:24.5789661Z","cpu":1.638,"memory":5324328.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.4435,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T16:58:34.5892073Z","cpu":4.134,"memory":5319308.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0977,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":39.5717,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":39.5676,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":39.540800000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":39.5385,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":38.5046,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T16:58:37.9338686Z","EndTimeUTC":"2026-04-14T16:58:37.9338686Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T16:58:37.9494770Z","DurationInMs":15.608400000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"dcbc993b-13d2-47fd-aa8f-2436509f499f","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.6","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 4:58:37 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T16:58:37.9338686Z", "durationInMs": 0.0074},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T16:58:37.9338760Z", "durationInMs": 17.6383},{"event": "Pipelined", "startTimeUtc": "2026-04-14T16:58:37.9515143Z", "durationInMs": 0.198},{"event": "Transit Time", "startTimeUtc": "2026-04-14T16:58:37.9517123Z", "durationInMs": 1.8388},{"event": "Received", "startTimeUtc": "2026-04-14T16:58:37.9535511Z", "durationInMs": 0.1444},{"event": "Completed", "startTimeUtc": "2026-04-14T16:58:37.9536955Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T16:58:37.9494770Z","lastSend":"2026-04-14T16:58:37.9494770Z","lastReceive":"2026-04-14T16:58:37.9494770Z"},"requestSizeInBytes":1635,"requestBodySizeInBytes":1183,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T16:58:37.9182936Z","DurationInMs":15.575000000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a4bd216a-1e73-41ec-9b08-5ae4b3c16dbf","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T16:58:37.9338686Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"89b50b95-16dc-4250-8cf9-19033157b313","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T16:58:37.9338686Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"dcbc993b-13d2-47fd-aa8f-2436509f499f","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.437509,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"dcbc993b-13d2-47fd-aa8f-2436509f499f","ResponseTimeUtc":"2026-04-14T16:58:37.9651035Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: dcbc993b-13d2-47fd-aa8f-2436509f499f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.6"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 17:01:21 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e892a9aa-56aa-48db-8780-4c49227d4596; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e892a9aa-56aa-48db-8780-4c49227d4596, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e892a9aa-56aa-48db-8780-4c49227d4596; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e892a9aa-56aa-48db-8780-4c49227d4596, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:01:21.581Z","duration in milliseconds":22.6451,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:01:21.2065207Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91451,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.042800000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":0.18810000000000002,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0019}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":22.236,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":22.1964,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:00:24.6842599Z","cpu":0.780,"memory":5224752.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1387,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:00:34.6909603Z","cpu":3.828,"memory":5236844.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0807,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:00:44.6952737Z","cpu":1.797,"memory":5229060.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0908,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:00:54.7046343Z","cpu":3.354,"memory":5224780.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1114,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:01:04.7205718Z","cpu":3.120,"memory":5225488.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0569,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:01:14.7287936Z","cpu":4.688,"memory":5228300.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1523,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":22.188200000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":22.183500000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":22.1657,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":22.1627,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":21.140700000000002,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:01:21.5971559Z","DurationInMs":15.5563,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"e892a9aa-56aa-48db-8780-4c49227d4596","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.716","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:01:21 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:01:21.5815996Z", "durationInMs": 0.0162},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:01:21.5816158Z", "durationInMs": 17.767},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:01:21.5993828Z", "durationInMs": 0.2018},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:01:21.5995846Z", "durationInMs": 2.2019},{"event": "Received", "startTimeUtc": "2026-04-14T17:01:21.6017865Z", "durationInMs": 0.1431},{"event": "Completed", "startTimeUtc": "2026-04-14T17:01:21.6019296Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:01:21.5971559Z","lastSend":"2026-04-14T17:01:21.5971559Z","lastReceive":"2026-04-14T17:01:21.5971559Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.39063519999999996,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"e892a9aa-56aa-48db-8780-4c49227d4596","ResponseTimeUtc":"2026-04-14T17:01:21.5971559Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: e892a9aa-56aa-48db-8780-4c49227d4596, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.716"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 17:07:08 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 43a35360-ede7-4de7-b6b8-407cff0e5787; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 43a35360-ede7-4de7-b6b8-407cff0e5787, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 43a35360-ede7-4de7-b6b8-407cff0e5787; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 43a35360-ede7-4de7-b6b8-407cff0e5787, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:07:08.131Z","duration in milliseconds":43.2545,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:07:07.9912034Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91527,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0516},{"name":"Get PkValue From Stream","duration in milliseconds":6.9227000000000007,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0016},{"name":"Read Collection","duration in milliseconds":6.7552,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:07:08.1318265Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"ea0beba4-7c80-4131-b23c-4977a2108b86","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.0888,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.0464,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:06:15.0479222Z","cpu":10.625,"memory":5144740.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.106,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:06:25.0610462Z","cpu":1.092,"memory":5140892.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2644,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:06:35.0677362Z","cpu":0.780,"memory":5152624.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1231,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:06:45.0775215Z","cpu":14.063,"memory":5174284.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2683,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:06:55.0832740Z","cpu":1.404,"memory":5192576.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0818,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:07:05.0991164Z","cpu":0.780,"memory":5190976.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1041,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.0372,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.0326,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.0098,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.0075,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.001000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:07:08.1475099Z","EndTimeUTC":"2026-04-14T17:07:08.1475099Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:07:08.1787023Z","DurationInMs":31.192400000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"43a35360-ede7-4de7-b6b8-407cff0e5787","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.806","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:07:08 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:07:08.1475099Z", "durationInMs": 0.0079},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:07:08.1475178Z", "durationInMs": 17.1363},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:07:08.1646541Z", "durationInMs": 0.2144},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:07:08.1648685Z", "durationInMs": 2.0411},{"event": "Received", "startTimeUtc": "2026-04-14T17:07:08.1669096Z", "durationInMs": 0.1351},{"event": "Completed", "startTimeUtc": "2026-04-14T17:07:08.1670447Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:07:08.1630778Z","lastSend":"2026-04-14T17:07:08.1630778Z","lastReceive":"2026-04-14T17:07:08.1630778Z"},"requestSizeInBytes":1642,"requestBodySizeInBytes":1192,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:07:08.1318265Z","DurationInMs":15.6834,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"938013d9-38d7-43a1-9731-b902b8cbe0fb","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:07:08.1475099Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"178423fe-bb1d-403f-a41d-614c7cc85da3","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:07:08.1475099Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"43a35360-ede7-4de7-b6b8-407cff0e5787","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1874989,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"43a35360-ede7-4de7-b6b8-407cff0e5787","ResponseTimeUtc":"2026-04-14T17:07:08.1787023Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 43a35360-ede7-4de7-b6b8-407cff0e5787, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.806"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0023}]}
2026-04-14 17:08:01 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: bf42e8a8-30c5-446f-91ae-653a32f14cb5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: bf42e8a8-30c5-446f-91ae-653a32f14cb5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: bf42e8a8-30c5-446f-91ae-653a32f14cb5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: bf42e8a8-30c5-446f-91ae-653a32f14cb5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:08:01.887Z","duration in milliseconds":66.5887,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:08:01.7627165Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91551,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0487},{"name":"Get PkValue From Stream","duration in milliseconds":7.7392,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":7.5695000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:08:01.8877157Z","DurationInMs":15.624500000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"697af8b8-c77c-4f1c-80e6-e89b28202943","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":58.595400000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":58.5536,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:07:05.0991164Z","cpu":0.780,"memory":5190976.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1041,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:07:15.1114492Z","cpu":3.125,"memory":5187240.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.107,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:07:25.1144334Z","cpu":9.360,"memory":5179696.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0981,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:07:35.1178707Z","cpu":10.000,"memory":5177000.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1002,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:07:45.1296835Z","cpu":4.368,"memory":5183900.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1078,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:07:55.1386012Z","cpu":6.797,"memory":5139980.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0909,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":58.543400000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":58.5386,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":58.512800000000006,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":58.510400000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":57.1627,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:08:01.9033402Z","EndTimeUTC":"2026-04-14T17:08:01.9189664Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:08:01.9502179Z","DurationInMs":31.2515,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"bf42e8a8-30c5-446f-91ae-653a32f14cb5","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.679","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:08:01 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:08:01.9189664Z", "durationInMs": 0.0079},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:08:01.9189743Z", "durationInMs": 18.6133},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:08:01.9375876Z", "durationInMs": 0.236},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:08:01.9378236Z", "durationInMs": 24.3287},{"event": "Received", "startTimeUtc": "2026-04-14T17:08:01.9621523Z", "durationInMs": 0.1812},{"event": "Completed", "startTimeUtc": "2026-04-14T17:08:01.9623335Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:08:01.9345906Z","lastSend":"2026-04-14T17:08:01.9345906Z","lastReceive":"2026-04-14T17:08:01.9345906Z"},"requestSizeInBytes":1644,"requestBodySizeInBytes":1190,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:08:01.9033402Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"004054cf-3ab9-4690-b1ab-272c4881be85","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:08:01.9033402Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"2ef5147e-2097-4988-aa58-91f81758a790","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:08:01.9033402Z","DurationInMs":15.6262,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"bf42e8a8-30c5-446f-91ae-653a32f14cb5","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2031238,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"bf42e8a8-30c5-446f-91ae-653a32f14cb5","ResponseTimeUtc":"2026-04-14T17:08:01.9658403Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: bf42e8a8-30c5-446f-91ae-653a32f14cb5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.679"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 17:09:08 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e5e0a7b4-bbfd-4469-9a13-f974e458da53; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e5e0a7b4-bbfd-4469-9a13-f974e458da53, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e5e0a7b4-bbfd-4469-9a13-f974e458da53; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e5e0a7b4-bbfd-4469-9a13-f974e458da53, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:09:07.973Z","duration in milliseconds":61.6974,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:09:07.8799464Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91566,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0555},{"name":"Get PkValue From Stream","duration in milliseconds":6.6383,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0043},{"name":"Read Collection","duration in milliseconds":6.4621,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:09:07.9736990Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"f006e37f-ef10-416a-a85e-ff00190adb2a","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":54.8134,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":54.7721,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:08:15.1552339Z","cpu":1.094,"memory":5183984.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0998,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:08:25.1674125Z","cpu":13.963,"memory":5250676.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2388,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:08:35.1694535Z","cpu":0.938,"memory":5246908.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1402,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:08:45.1814853Z","cpu":6.084,"memory":5193800.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1028,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:08:55.1837194Z","cpu":2.188,"memory":5269724.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1008,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:09:05.1924003Z","cpu":2.340,"memory":5259684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1228,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":54.762100000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":54.7584,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":54.735800000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":54.733200000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":53.7256,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:09:08.0049494Z","EndTimeUTC":"2026-04-14T17:09:08.0049494Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:09:08.0362023Z","DurationInMs":31.2529,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"e5e0a7b4-bbfd-4469-9a13-f974e458da53","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.599","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:09:08 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:09:08.0049494Z", "durationInMs": 0.0072},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:09:08.0049566Z", "durationInMs": 17.718},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:09:08.0226746Z", "durationInMs": 0.2498},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:09:08.0229244Z", "durationInMs": 2.1071},{"event": "Received", "startTimeUtc": "2026-04-14T17:09:08.0250315Z", "durationInMs": 0.1554},{"event": "Completed", "startTimeUtc": "2026-04-14T17:09:08.0251869Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:09:08.0205740Z","lastSend":"2026-04-14T17:09:08.0205740Z","lastReceive":"2026-04-14T17:09:08.0362023Z"},"requestSizeInBytes":1642,"requestBodySizeInBytes":1192,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:09:07.9736990Z","DurationInMs":15.6264,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"576ab8fa-dbfc-4a1e-8e03-eec403554bd8","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:09:08.0049494Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"9d26ba27-58f5-44d1-a341-4c8ffb6357bd","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:09:08.0049494Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"e5e0a7b4-bbfd-4469-9a13-f974e458da53","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1562559,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"e5e0a7b4-bbfd-4469-9a13-f974e458da53","ResponseTimeUtc":"2026-04-14T17:09:08.0362023Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: e5e0a7b4-bbfd-4469-9a13-f974e458da53, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.599"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 17:10:10 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 7707f675-a303-4558-b3e4-dd3d7be7184d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 7707f675-a303-4558-b3e4-dd3d7be7184d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 7707f675-a303-4558-b3e4-dd3d7be7184d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 7707f675-a303-4558-b3e4-dd3d7be7184d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:10:10.561Z","duration in milliseconds":45.2308,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:10:10.4829966Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91583,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.054200000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":6.2248,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0044},{"name":"Read Collection","duration in milliseconds":6.054,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:10:10.5611258Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"b9cfa065-f147-45cd-b9a7-9cfb80691b22","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":38.7693,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":38.731700000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:09:15.2021070Z","cpu":2.109,"memory":5256336.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0868,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:09:25.2193990Z","cpu":3.588,"memory":5253420.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1889,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:09:35.2205623Z","cpu":1.016,"memory":5251056.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1124,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:09:45.2304311Z","cpu":3.042,"memory":5253592.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1006,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:09:55.2370719Z","cpu":1.641,"memory":5250724.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1599,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:10:05.2401368Z","cpu":1.872,"memory":5231840.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1078,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":38.724000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":38.7197,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":38.6948,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":38.6914,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":37.8241,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:10:10.5768096Z","EndTimeUTC":"2026-04-14T17:10:10.5768096Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:10:10.6080029Z","DurationInMs":31.1933,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"7707f675-a303-4558-b3e4-dd3d7be7184d","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.585","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:10:10 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:10:10.5768096Z", "durationInMs": 0.0095},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:10:10.5768191Z", "durationInMs": 17.8464},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:10:10.5946655Z", "durationInMs": 0.2183},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:10:10.5948838Z", "durationInMs": 1.7971},{"event": "Received", "startTimeUtc": "2026-04-14T17:10:10.5966809Z", "durationInMs": 0.1425},{"event": "Completed", "startTimeUtc": "2026-04-14T17:10:10.5968234Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:10:10.5923769Z","lastSend":"2026-04-14T17:10:10.5923769Z","lastReceive":"2026-04-14T17:10:10.6080029Z"},"requestSizeInBytes":1648,"requestBodySizeInBytes":1192,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:10:10.5611258Z","DurationInMs":15.683800000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"309901e2-3777-43a5-842d-eacbb88dfdbc","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:10:10.5768096Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"3fb0c2ee-bdb5-42ea-8e9d-9a9ed9e1a96e","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:10:10.5768096Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"7707f675-a303-4558-b3e4-dd3d7be7184d","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1250063,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"7707f675-a303-4558-b3e4-dd3d7be7184d","ResponseTimeUtc":"2026-04-14T17:10:10.6080029Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 7707f675-a303-4558-b3e4-dd3d7be7184d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.585"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 17:11:31 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 3a1da119-2748-4460-9894-59f6e7f1b5d9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 3a1da119-2748-4460-9894-59f6e7f1b5d9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 3a1da119-2748-4460-9894-59f6e7f1b5d9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 3a1da119-2748-4460-9894-59f6e7f1b5d9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:11:31.840Z","duration in milliseconds":42.5381,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:11:31.7305503Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91606,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.047400000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":6.5298000000000007,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0058000000000000005},{"name":"Read Collection","duration in milliseconds":6.3676,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:11:31.8400036Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"140d6f07-6a71-4b1f-a8a9-4b26f740a149","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.7828,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.7523,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:10:35.2588369Z","cpu":1.094,"memory":5180776.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0679,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:10:45.2734042Z","cpu":1.560,"memory":5192616.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1027,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:10:55.2825576Z","cpu":1.797,"memory":5190440.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0855,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:11:05.2928153Z","cpu":6.474,"memory":5187948.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.115,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:11:15.2979780Z","cpu":1.875,"memory":5194592.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0697,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:11:25.3120043Z","cpu":2.886,"memory":5194032.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0864,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.7449,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.740300000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.7202,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.7179,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.744800000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:11:31.8556099Z","EndTimeUTC":"2026-04-14T17:11:31.8556099Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:11:31.8711779Z","DurationInMs":15.568000000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"3a1da119-2748-4460-9894-59f6e7f1b5d9","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.547","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:11:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:11:31.8556099Z", "durationInMs": 0.0093},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:11:31.8556192Z", "durationInMs": 17.7344},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:11:31.8733536Z", "durationInMs": 0.1909},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:11:31.8735445Z", "durationInMs": 1.8114},{"event": "Received", "startTimeUtc": "2026-04-14T17:11:31.8753559Z", "durationInMs": 0.1301},{"event": "Completed", "startTimeUtc": "2026-04-14T17:11:31.8754860Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:11:31.8711779Z","lastSend":"2026-04-14T17:11:31.8711779Z","lastReceive":"2026-04-14T17:11:31.8711779Z"},"requestSizeInBytes":1640,"requestBodySizeInBytes":1192,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:11:31.8400036Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"4981a844-353b-4116-834e-320793bd070e","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:11:31.8400036Z","DurationInMs":15.606300000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"66d18a13-0c53-4241-8218-d3c1e084ebe8","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:11:31.8556099Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"3a1da119-2748-4460-9894-59f6e7f1b5d9","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1406276,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"3a1da119-2748-4460-9894-59f6e7f1b5d9","ResponseTimeUtc":"2026-04-14T17:11:31.8711779Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 3a1da119-2748-4460-9894-59f6e7f1b5d9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.547"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 17:12:16 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 0009760a-7611-42c3-b776-adf5c69b4a00; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 0009760a-7611-42c3-b776-adf5c69b4a00, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 0009760a-7611-42c3-b776-adf5c69b4a00; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 0009760a-7611-42c3-b776-adf5c69b4a00, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:12:16.235Z","duration in milliseconds":48.8742,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:12:16.1105121Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91610,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.055200000000000006},{"name":"Get PkValue From Stream","duration in milliseconds":6.4361000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0042000000000000006},{"name":"Read Collection","duration in milliseconds":6.2507,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:12:16.2355972Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"6bf88258-74ce-4a1e-9085-15f20297f88d","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":42.0903,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":42.0431,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:11:25.3120043Z","cpu":2.886,"memory":5194032.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0864,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:11:35.3166327Z","cpu":1.406,"memory":5193396.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1233,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:11:45.3306587Z","cpu":1.872,"memory":5193852.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1098,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:11:55.3450842Z","cpu":0.546,"memory":5196760.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1102,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:12:05.3566787Z","cpu":1.326,"memory":5198452.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0875,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:12:15.3605935Z","cpu":0.547,"memory":5199800.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0931,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":42.0336,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":42.028,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":42.0066,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":42.004400000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":40.911,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:12:16.2511431Z","EndTimeUTC":"2026-04-14T17:12:16.2511431Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:12:16.2824560Z","DurationInMs":31.312900000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"0009760a-7611-42c3-b776-adf5c69b4a00","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.586","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:12:16 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:12:16.2511431Z", "durationInMs": 0.0098},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:12:16.2511529Z", "durationInMs": 21.5161},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:12:16.2726690Z", "durationInMs": 0.2152},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:12:16.2728842Z", "durationInMs": 4.9586},{"event": "Received", "startTimeUtc": "2026-04-14T17:12:16.2778428Z", "durationInMs": 0.1599},{"event": "Completed", "startTimeUtc": "2026-04-14T17:12:16.2780027Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:12:16.2667774Z","lastSend":"2026-04-14T17:12:16.2667774Z","lastReceive":"2026-04-14T17:12:16.2824560Z"},"requestSizeInBytes":1640,"requestBodySizeInBytes":1190,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:12:16.2355972Z","DurationInMs":15.545900000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"c0d3628b-bc45-4907-8f6e-2a67ce578e3f","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:12:16.2511431Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"1ab440b2-4b76-4476-805f-89a1ff3a3195","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:12:16.2511431Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"0009760a-7611-42c3-b776-adf5c69b4a00","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.17194389999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"0009760a-7611-42c3-b776-adf5c69b4a00","ResponseTimeUtc":"2026-04-14T17:12:16.2824560Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 0009760a-7611-42c3-b776-adf5c69b4a00, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.586"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0014}]}
2026-04-14 17:19:49 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 919a8071-a3f0-41f0-a1f6-4ff9cedd3e1d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 919a8071-a3f0-41f0-a1f6-4ff9cedd3e1d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 919a8071-a3f0-41f0-a1f6-4ff9cedd3e1d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 919a8071-a3f0-41f0-a1f6-4ff9cedd3e1d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:19:49.932Z","duration in milliseconds":22.0881,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:19:49.7602927Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91652,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.039},{"name":"Get PkValue From Stream","duration in milliseconds":0.1544,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":21.651300000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":21.613500000000002,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:18:55.6471577Z","cpu":1.797,"memory":5283756.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2629,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:19:05.6540540Z","cpu":1.250,"memory":5283012.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0747,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:19:15.6669489Z","cpu":2.730,"memory":5282296.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2631,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:19:25.6828965Z","cpu":1.794,"memory":5284256.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1189,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:19:35.6863569Z","cpu":0.938,"memory":5279864.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0705,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:19:45.6915249Z","cpu":2.106,"memory":5276036.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0793,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":21.6062,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":21.6016,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":21.584,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":21.581300000000002,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":20.5883,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:19:49.9477981Z","DurationInMs":15.626600000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"919a8071-a3f0-41f0-a1f6-4ff9cedd3e1d","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.829","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:19:49 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:19:49.9321715Z", "durationInMs": 0.019},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:19:49.9321905Z", "durationInMs": 17.476},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:19:49.9496665Z", "durationInMs": 0.1918},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:19:49.9498583Z", "durationInMs": 1.9326},{"event": "Received", "startTimeUtc": "2026-04-14T17:19:49.9517909Z", "durationInMs": 0.1468},{"event": "Completed", "startTimeUtc": "2026-04-14T17:19:49.9519377Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:19:49.9477981Z","lastSend":"2026-04-14T17:19:49.9477981Z","lastReceive":"2026-04-14T17:19:49.9477981Z"},"requestSizeInBytes":1635,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.1875054,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"919a8071-a3f0-41f0-a1f6-4ff9cedd3e1d","ResponseTimeUtc":"2026-04-14T17:19:49.9477981Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 919a8071-a3f0-41f0-a1f6-4ff9cedd3e1d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.829"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 17:20:08 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 2a816b3f-a9c6-40f9-99d6-5d52327d4416; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 2a816b3f-a9c6-40f9-99d6-5d52327d4416, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 2a816b3f-a9c6-40f9-99d6-5d52327d4416; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 2a816b3f-a9c6-40f9-99d6-5d52327d4416, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:20:07.965Z","duration in milliseconds":45.2062,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:20:07.6525069Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91654,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0373},{"name":"Get PkValue From Stream","duration in milliseconds":9.5774,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.0316,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:20:07.9832514Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"86cdc162-a0dd-48c7-b497-5bafe49a1d84","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.4155,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.382000000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:19:15.6669489Z","cpu":2.730,"memory":5282296.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2631,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:19:25.6828965Z","cpu":1.794,"memory":5284256.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1189,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:19:35.6863569Z","cpu":0.938,"memory":5279864.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0705,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:19:45.6915249Z","cpu":2.106,"memory":5276036.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0793,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:19:55.6965772Z","cpu":1.797,"memory":5273480.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0753,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:20:05.6994447Z","cpu":2.031,"memory":5258432.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1683,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.3751,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.368700000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.349900000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.347300000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.487700000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:20:07.9963162Z","EndTimeUTC":"2026-04-14T17:20:07.9963162Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:20:08.0118859Z","DurationInMs":15.569700000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"2a816b3f-a9c6-40f9-99d6-5d52327d4416","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.661","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:20:08 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:20:07.9963162Z", "durationInMs": 0.008},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:20:07.9963242Z", "durationInMs": 18.1794},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:20:08.0145036Z", "durationInMs": 0.2038},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:20:08.0147074Z", "durationInMs": 1.9505},{"event": "Received", "startTimeUtc": "2026-04-14T17:20:08.0166579Z", "durationInMs": 0.1469},{"event": "Completed", "startTimeUtc": "2026-04-14T17:20:08.0168048Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:20:08.0118859Z","lastSend":"2026-04-14T17:20:08.0118859Z","lastReceive":"2026-04-14T17:20:08.0118859Z"},"requestSizeInBytes":1608,"requestBodySizeInBytes":1156,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:20:07.9832514Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"6e357aaf-10f3-468d-a323-f9cb813a288c","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:20:07.9832514Z","DurationInMs":13.0648,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"962ef1a5-558f-4ce7-a829-8df3dd36b05c","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:20:07.9963162Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"2a816b3f-a9c6-40f9-99d6-5d52327d4416","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.359379,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"2a816b3f-a9c6-40f9-99d6-5d52327d4416","ResponseTimeUtc":"2026-04-14T17:20:08.0118859Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 2a816b3f-a9c6-40f9-99d6-5d52327d4416, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.661"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 17:25:46 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e68c7d69-823b-47ee-b39f-99153ac77eb0; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e68c7d69-823b-47ee-b39f-99153ac77eb0, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e68c7d69-823b-47ee-b39f-99153ac77eb0; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e68c7d69-823b-47ee-b39f-99153ac77eb0, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:25:46.141Z","duration in milliseconds":51.266600000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:25:45.4693282Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91681,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0536},{"name":"Get PkValue From Stream","duration in milliseconds":7.5944,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002},{"name":"Read Collection","duration in milliseconds":7.4401,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:25:46.1412086Z","DurationInMs":15.7521,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"37160842-3d1d-4b27-9342-7f341186a40c","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":43.4333,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":43.3996,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:24:55.9833746Z","cpu":1.950,"memory":5257692.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0963,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:25:05.9965429Z","cpu":3.978,"memory":5226924.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1049,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:25:16.0096489Z","cpu":2.496,"memory":5224828.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0948,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:25:26.0190754Z","cpu":7.500,"memory":5224284.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.18,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:25:36.0290738Z","cpu":4.836,"memory":5226404.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1124,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:25:46.0318307Z","cpu":1.875,"memory":5226576.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0921,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":43.3918,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":43.3877,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":43.3659,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":43.3634,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":42.3825,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:25:46.1725117Z","EndTimeUTC":"2026-04-14T17:25:46.1725117Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:25:46.2037075Z","DurationInMs":31.195800000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"e68c7d69-823b-47ee-b39f-99153ac77eb0","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.82","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:25:46 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:25:46.1725117Z", "durationInMs": 0.0096},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:25:46.1725213Z", "durationInMs": 20.4184},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:25:46.1929397Z", "durationInMs": 0.2177},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:25:46.1931574Z", "durationInMs": 1.9905},{"event": "Received", "startTimeUtc": "2026-04-14T17:25:46.1951479Z", "durationInMs": 0.153},{"event": "Completed", "startTimeUtc": "2026-04-14T17:25:46.1953009Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:25:46.1880828Z","lastSend":"2026-04-14T17:25:46.1880828Z","lastReceive":"2026-04-14T17:25:46.1880828Z"},"requestSizeInBytes":1606,"requestBodySizeInBytes":1156,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:25:46.1569607Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"739db55a-bea1-45bf-832f-3dd1cdecd32f","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:25:46.1569607Z","DurationInMs":15.551,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"15b05d03-02fa-44c4-a7f3-da5602aa9a4d","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:25:46.1725117Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"e68c7d69-823b-47ee-b39f-99153ac77eb0","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.7343793,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"e68c7d69-823b-47ee-b39f-99153ac77eb0","ResponseTimeUtc":"2026-04-14T17:25:46.2037075Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: e68c7d69-823b-47ee-b39f-99153ac77eb0, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.82"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 17:25:53 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 2f21873b-0587-4b54-b998-e1b6bceec6a2; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 2f21873b-0587-4b54-b998-e1b6bceec6a2, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 2f21873b-0587-4b54-b998-e1b6bceec6a2; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 2f21873b-0587-4b54-b998-e1b6bceec6a2, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:25:53.347Z","duration in milliseconds":39.7322,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:25:52.2379719Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91682,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.043300000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":6.1085,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0037},{"name":"Read Collection","duration in milliseconds":5.9553,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:25:53.3473797Z","DurationInMs":15.6651,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"55ea5803-f1d7-4810-8a68-5570f3db172e","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":33.4166,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":33.3865,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:24:55.9833746Z","cpu":1.950,"memory":5257692.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0963,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:25:05.9965429Z","cpu":3.978,"memory":5226924.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1049,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:25:16.0096489Z","cpu":2.496,"memory":5224828.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0948,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:25:26.0190754Z","cpu":7.500,"memory":5224284.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.18,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:25:36.0290738Z","cpu":4.836,"memory":5226404.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1124,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:25:46.0318307Z","cpu":1.875,"memory":5226576.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0921,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":33.3788,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":33.374500000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":33.3559,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":33.3536,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":32.403400000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:25:53.3630448Z","EndTimeUTC":"2026-04-14T17:25:53.3787209Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:25:53.3942492Z","DurationInMs":15.528300000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"2f21873b-0587-4b54-b998-e1b6bceec6a2","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.523","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:25:53 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:25:53.3787209Z", "durationInMs": 0.008},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:25:53.3787289Z", "durationInMs": 15.1345},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:25:53.3938634Z", "durationInMs": 0.1986},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:25:53.3940620Z", "durationInMs": 1.7139},{"event": "Received", "startTimeUtc": "2026-04-14T17:25:53.3957759Z", "durationInMs": 0.1274},{"event": "Completed", "startTimeUtc": "2026-04-14T17:25:53.3959033Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:25:53.3942492Z","lastSend":"2026-04-14T17:25:53.3942492Z","lastReceive":"2026-04-14T17:25:53.3942492Z"},"requestSizeInBytes":1641,"requestBodySizeInBytes":1189,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:25:53.3630448Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a196e187-4dc8-4eec-9530-e33b0ca70ff3","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:25:53.3630448Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"ac9b591a-6872-40db-8992-45e9b5b35284","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:25:53.3630448Z","DurationInMs":15.6761,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"2f21873b-0587-4b54-b998-e1b6bceec6a2","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":1.1562773,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"2f21873b-0587-4b54-b998-e1b6bceec6a2","ResponseTimeUtc":"2026-04-14T17:25:53.3942492Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 2f21873b-0587-4b54-b998-e1b6bceec6a2, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.523"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 17:27:49 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 59c9b25e-7120-4b20-bc2c-baedd89e306d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 59c9b25e-7120-4b20-bc2c-baedd89e306d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 59c9b25e-7120-4b20-bc2c-baedd89e306d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 59c9b25e-7120-4b20-bc2c-baedd89e306d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:27:49.402Z","duration in milliseconds":58.4256,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:27:49.1051981Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91692,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.047900000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":7.3247,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0013000000000000002},{"name":"Read Collection","duration in milliseconds":7.1695,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:27:49.4020864Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"47d00023-14c7-44ee-a8ef-35f8a5e9200d","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":50.8714,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":50.8367,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:26:56.0795457Z","cpu":2.340,"memory":5184712.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0898,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:27:06.0895265Z","cpu":0.546,"memory":5186752.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0948,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:27:16.0975689Z","cpu":2.109,"memory":5186656.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1036,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:27:26.1072537Z","cpu":0.936,"memory":5186604.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1276,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:27:36.1176027Z","cpu":1.094,"memory":5186224.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1112,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:27:46.1223480Z","cpu":0.858,"memory":5195448.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2625,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":50.829100000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":50.8252,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":50.804100000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":50.8018,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":49.9086,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:27:49.4333836Z","EndTimeUTC":"2026-04-14T17:27:49.4333836Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:27:49.4489503Z","DurationInMs":15.5667,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"59c9b25e-7120-4b20-bc2c-baedd89e306d","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.587","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:27:49 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:27:49.4333836Z", "durationInMs": 0.0097},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:27:49.4333933Z", "durationInMs": 21.8472},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:27:49.4552405Z", "durationInMs": 0.1992},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:27:49.4554397Z", "durationInMs": 2.004},{"event": "Received", "startTimeUtc": "2026-04-14T17:27:49.4574437Z", "durationInMs": 0.1476},{"event": "Completed", "startTimeUtc": "2026-04-14T17:27:49.4575913Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:27:49.4489503Z","lastSend":"2026-04-14T17:27:49.4489503Z","lastReceive":"2026-04-14T17:27:49.4489503Z"},"requestSizeInBytes":1604,"requestBodySizeInBytes":1156,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:27:49.4020864Z","DurationInMs":15.6759,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"ae983ee2-f9fe-4a65-b43c-07eaf6908724","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:27:49.4177623Z","DurationInMs":15.621300000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"1b5ea443-7178-431f-aee2-75e518006f5b","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:27:49.4333836Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"59c9b25e-7120-4b20-bc2c-baedd89e306d","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.3437522,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"59c9b25e-7120-4b20-bc2c-baedd89e306d","ResponseTimeUtc":"2026-04-14T17:27:49.4645822Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 59c9b25e-7120-4b20-bc2c-baedd89e306d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.587"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0025}]}
2026-04-14 17:30:01 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 462d68af-b560-4a69-9c5e-eac9e82e533e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 462d68af-b560-4a69-9c5e-eac9e82e533e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 462d68af-b560-4a69-9c5e-eac9e82e533e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 462d68af-b560-4a69-9c5e-eac9e82e533e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:30:01.205Z","duration in milliseconds":44.6637,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:30:01.0807126Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91744,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.051800000000000006},{"name":"Get PkValue From Stream","duration in milliseconds":6.8016000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002},{"name":"Read Collection","duration in milliseconds":6.6202000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:30:01.2057170Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"48c5e6cc-3f8c-440e-9e6b-1b2f1e1faef9","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.6066,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.5653,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:29:06.1855545Z","cpu":3.978,"memory":5281460.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.107,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:29:16.1996770Z","cpu":3.744,"memory":5304072.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0951,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:29:26.2040068Z","cpu":6.406,"memory":5298788.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0967,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:29:36.2170986Z","cpu":11.388,"memory":5293308.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1164,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:29:46.2187825Z","cpu":8.750,"memory":5296568.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.046,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:29:56.2195880Z","cpu":2.578,"memory":5293660.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0662,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.5538,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.5507,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.5278,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.5253,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.5003,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:30:01.2213461Z","EndTimeUTC":"2026-04-14T17:30:01.2213461Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:30:01.2525914Z","DurationInMs":31.2453,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"462d68af-b560-4a69-9c5e-eac9e82e533e","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.595","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:30:01 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:30:01.2213461Z", "durationInMs": 0.0078},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:30:01.2213539Z", "durationInMs": 19.0986},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:30:01.2404525Z", "durationInMs": 0.2377},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:30:01.2406902Z", "durationInMs": 2.0597},{"event": "Received", "startTimeUtc": "2026-04-14T17:30:01.2427499Z", "durationInMs": 0.173},{"event": "Completed", "startTimeUtc": "2026-04-14T17:30:01.2429229Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:30:01.2369682Z","lastSend":"2026-04-14T17:30:01.2369682Z","lastReceive":"2026-04-14T17:30:01.2369682Z"},"requestSizeInBytes":1630,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:30:01.2057170Z","DurationInMs":15.629100000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"806f1b43-ef6e-43c6-b691-0ffcf3f8fe6a","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:30:01.2213461Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0679fe4a-7e5c-4d7b-94da-d89a80f9b224","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:30:01.2213461Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"462d68af-b560-4a69-9c5e-eac9e82e533e","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718788,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"462d68af-b560-4a69-9c5e-eac9e82e533e","ResponseTimeUtc":"2026-04-14T17:30:01.2525914Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 462d68af-b560-4a69-9c5e-eac9e82e533e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.595"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 17:30:50 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 216a63bd-68a5-4605-88c7-00d5232cea99; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 216a63bd-68a5-4605-88c7-00d5232cea99, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 216a63bd-68a5-4605-88c7-00d5232cea99; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 216a63bd-68a5-4605-88c7-00d5232cea99, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:30:50.290Z","duration in milliseconds":4.9181,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:30:49.6808476Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91776,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0506},{"name":"Get PkValue From Stream","duration in milliseconds":0.1647,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0023}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":4.5273,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":4.4953,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:29:56.2195880Z","cpu":2.578,"memory":5293660.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0662,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:30:06.2319859Z","cpu":11.778,"memory":5272828.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1036,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1},{"dateUtc":"2026-04-14T17:30:16.2443120Z","cpu":4.844,"memory":5226312.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0335,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:30:26.2478395Z","cpu":2.808,"memory":5223884.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0976,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:30:36.2610841Z","cpu":3.359,"memory":5223080.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1087,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:30:46.2764165Z","cpu":3.042,"memory":5234336.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0463,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":4.4883000000000006,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":4.4833,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":4.4671,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":4.4647000000000006,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":3.3892,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:30:50.2903037Z","DurationInMs":0,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"216a63bd-68a5-4605-88c7-00d5232cea99","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.549","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:30:50 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:30:50.2903037Z", "durationInMs": 0.0033},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:30:50.2903070Z", "durationInMs": 0.0034},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:30:50.2903104Z", "durationInMs": 0.2315},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:30:50.2905419Z", "durationInMs": 2.3085},{"event": "Received", "startTimeUtc": "2026-04-14T17:30:50.2928504Z", "durationInMs": 0.0777},{"event": "Completed", "startTimeUtc": "2026-04-14T17:30:50.2929281Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:30:50.2589830Z","lastSend":"2026-04-14T17:30:50.2589830Z","lastReceive":"2026-04-14T17:30:50.2589830Z"},"requestSizeInBytes":1629,"requestBodySizeInBytes":1181,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.6094561,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"216a63bd-68a5-4605-88c7-00d5232cea99","ResponseTimeUtc":"2026-04-14T17:30:50.2903037Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 216a63bd-68a5-4605-88c7-00d5232cea99, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.549"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001}]}
2026-04-14 17:30:55 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c660f7ea-3556-4f61-ad6a-8853c2d9f7f7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c660f7ea-3556-4f61-ad6a-8853c2d9f7f7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c660f7ea-3556-4f61-ad6a-8853c2d9f7f7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c660f7ea-3556-4f61-ad6a-8853c2d9f7f7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:30:55.832Z","duration in milliseconds":41.967400000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:30:55.7229282Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91777,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.048100000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":6.1339000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":5.9749,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:30:55.8323140Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"7e63f1b9-e87d-44e6-a855-70291035fb80","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.6032,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.5657,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:29:56.2195880Z","cpu":2.578,"memory":5293660.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0662,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:30:06.2319859Z","cpu":11.778,"memory":5272828.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1036,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1},{"dateUtc":"2026-04-14T17:30:16.2443120Z","cpu":4.844,"memory":5226312.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0335,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:30:26.2478395Z","cpu":2.808,"memory":5223884.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0976,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:30:36.2610841Z","cpu":3.359,"memory":5223080.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1087,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:30:46.2764165Z","cpu":3.042,"memory":5234336.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0463,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.557300000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.5516,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.5302,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.5277,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.5489,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:30:55.8479906Z","EndTimeUTC":"2026-04-14T17:30:55.8479906Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:30:55.8635566Z","DurationInMs":15.566,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c660f7ea-3556-4f61-ad6a-8853c2d9f7f7","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.537","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:30:55 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:30:55.8479906Z", "durationInMs": 0.0083},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:30:55.8479989Z", "durationInMs": 17.2697},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:30:55.8652686Z", "durationInMs": 0.2058},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:30:55.8654744Z", "durationInMs": 2.1411},{"event": "Received", "startTimeUtc": "2026-04-14T17:30:55.8676155Z", "durationInMs": 0.1512},{"event": "Completed", "startTimeUtc": "2026-04-14T17:30:55.8677667Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:30:55.8635566Z","lastSend":"2026-04-14T17:30:55.8635566Z","lastReceive":"2026-04-14T17:30:55.8635566Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:30:55.8323140Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"48b05d9e-8d0a-49c3-a1fe-82b8b921bd10","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:30:55.8323140Z","DurationInMs":15.6766,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"b1519615-2522-4f52-9c32-351ed7574cd5","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:30:55.8479906Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"c660f7ea-3556-4f61-ad6a-8853c2d9f7f7","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1406284,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c660f7ea-3556-4f61-ad6a-8853c2d9f7f7","ResponseTimeUtc":"2026-04-14T17:30:55.8635566Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c660f7ea-3556-4f61-ad6a-8853c2d9f7f7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.537"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0024000000000000002}]}
2026-04-14 17:31:09 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 03a41732-6168-4b90-9b6c-948743da97c6; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 03a41732-6168-4b90-9b6c-948743da97c6, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 03a41732-6168-4b90-9b6c-948743da97c6; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 03a41732-6168-4b90-9b6c-948743da97c6, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:31:09.361Z","duration in milliseconds":60.7436,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:31:09.2361841Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91780,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.042800000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":5.9084,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0013000000000000002},{"name":"Read Collection","duration in milliseconds":5.7623000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:31:09.3612633Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"044ea63c-1217-4d1e-9e1a-08f5d6239e5c","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":54.621300000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":54.5885,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:30:16.2443120Z","cpu":4.844,"memory":5226312.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0335,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:30:26.2478395Z","cpu":2.808,"memory":5223884.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0976,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:30:36.2610841Z","cpu":3.359,"memory":5223080.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1087,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:30:46.2764165Z","cpu":3.042,"memory":5234336.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0463,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:30:56.2854605Z","cpu":4.056,"memory":5229136.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0913,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:31:06.2976933Z","cpu":2.886,"memory":5228412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1023,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":54.5808,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":54.5773,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":54.558400000000006,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":54.5551,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":53.6924,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:31:09.3768169Z","EndTimeUTC":"2026-04-14T17:31:09.3768169Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:31:09.4236892Z","DurationInMs":46.8723,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"03a41732-6168-4b90-9b6c-948743da97c6","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149539,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149539,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149539,"SessionToken":"-1#26149539","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.645","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:31:09 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:31:09.3768169Z", "durationInMs": 0.0098},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:31:09.3768267Z", "durationInMs": 37.5553},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:31:09.4143820Z", "durationInMs": 0.243},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:31:09.4146250Z", "durationInMs": 1.7722},{"event": "Received", "startTimeUtc": "2026-04-14T17:31:09.4163972Z", "durationInMs": 0.1521},{"event": "Completed", "startTimeUtc": "2026-04-14T17:31:09.4165493Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:31:09.4080755Z","lastSend":"2026-04-14T17:31:09.4080755Z","lastReceive":"2026-04-14T17:31:09.4080755Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:31:09.3612633Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a717159a-6b04-4d8d-8abc-1f0157bd6df8","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:31:09.3612633Z","DurationInMs":15.553600000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"5f172eb2-52f7-4b09-bb9d-153f8f1983a1","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:31:09.3768169Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"03a41732-6168-4b90-9b6c-948743da97c6","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.18750509999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"03a41732-6168-4b90-9b6c-948743da97c6","ResponseTimeUtc":"2026-04-14T17:31:09.4236892Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 03a41732-6168-4b90-9b6c-948743da97c6, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149539","BELatencyInMs":"0.645"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0014}]}
2026-04-14 17:31:44 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 596bb66a-1424-419d-83c4-0ba89bf8cef1; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 596bb66a-1424-419d-83c4-0ba89bf8cef1, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 596bb66a-1424-419d-83c4-0ba89bf8cef1; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 596bb66a-1424-419d-83c4-0ba89bf8cef1, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:31:44.080Z","duration in milliseconds":47.287,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:31:43.9714890Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91795,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.055900000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":6.8655,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0027},{"name":"Read Collection","duration in milliseconds":6.6929000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:31:44.0809168Z","DurationInMs":15.6454,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"ea1a5ef7-88d1-494d-9650-187a846a4177","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":40.167,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":40.1261,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:30:46.2764165Z","cpu":3.042,"memory":5234336.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0463,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:30:56.2854605Z","cpu":4.056,"memory":5229136.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0913,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:31:06.2976933Z","cpu":2.886,"memory":5228412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1023,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:31:16.3050833Z","cpu":2.813,"memory":5231136.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1463,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:31:26.3134507Z","cpu":2.574,"memory":5230192.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1265,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:31:36.3182400Z","cpu":4.844,"memory":5231332.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1423,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":40.1169,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":40.1126,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":40.0903,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":40.087700000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":39.0764,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:31:44.0965622Z","EndTimeUTC":"2026-04-14T17:31:44.1121595Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:31:44.1277418Z","DurationInMs":15.5823,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"596bb66a-1424-419d-83c4-0ba89bf8cef1","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.779","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:31:44 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:31:44.1121595Z", "durationInMs": 0.0086},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:31:44.1121681Z", "durationInMs": 19.7533},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:31:44.1319214Z", "durationInMs": 0.2056},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:31:44.1321270Z", "durationInMs": 2.4249},{"event": "Received", "startTimeUtc": "2026-04-14T17:31:44.1345519Z", "durationInMs": 0.1615},{"event": "Completed", "startTimeUtc": "2026-04-14T17:31:44.1347134Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:31:44.1277418Z","lastSend":"2026-04-14T17:31:44.1277418Z","lastReceive":"2026-04-14T17:31:44.1277418Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:31:44.0965622Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"121e553f-0445-4928-b506-0b75f634bfff","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:31:44.0965622Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"6f3c2edf-fed1-4a38-b8ec-2e9c6808e980","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:31:44.0965622Z","DurationInMs":15.5973,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"596bb66a-1424-419d-83c4-0ba89bf8cef1","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.17187449999999999,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"596bb66a-1424-419d-83c4-0ba89bf8cef1","ResponseTimeUtc":"2026-04-14T17:31:44.1277418Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 596bb66a-1424-419d-83c4-0ba89bf8cef1, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.779"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 17:31:55 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: cb5ed8cd-03aa-4810-b50f-adcfd9c5f157; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: cb5ed8cd-03aa-4810-b50f-adcfd9c5f157, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: cb5ed8cd-03aa-4810-b50f-adcfd9c5f157; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: cb5ed8cd-03aa-4810-b50f-adcfd9c5f157, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:31:55.352Z","duration in milliseconds":42.865100000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:31:55.2741883Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91797,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0461},{"name":"Get PkValue From Stream","duration in milliseconds":6.1128,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":5.9545,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:31:55.3523106Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"05abcde3-f656-4670-a486-06e33ac248ba","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.534800000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.499300000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:30:56.2854605Z","cpu":4.056,"memory":5229136.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0913,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:31:06.2976933Z","cpu":2.886,"memory":5228412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1023,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:31:16.3050833Z","cpu":2.813,"memory":5231136.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1463,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:31:26.3134507Z","cpu":2.574,"memory":5230192.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1265,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:31:36.3182400Z","cpu":4.844,"memory":5231332.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1423,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:31:46.3248268Z","cpu":2.734,"memory":5231692.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1362,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.4926,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.4883,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.467600000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.4647,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.606700000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:31:55.3680103Z","EndTimeUTC":"2026-04-14T17:31:55.3680103Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:31:55.3992240Z","DurationInMs":31.213700000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"cb5ed8cd-03aa-4810-b50f-adcfd9c5f157","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.515","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:31:55 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:31:55.3680103Z", "durationInMs": 0.0092},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:31:55.3680195Z", "durationInMs": 17.7425},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:31:55.3857620Z", "durationInMs": 0.1952},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:31:55.3859572Z", "durationInMs": 2.0034},{"event": "Received", "startTimeUtc": "2026-04-14T17:31:55.3879606Z", "durationInMs": 0.127},{"event": "Completed", "startTimeUtc": "2026-04-14T17:31:55.3880876Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:31:55.3836300Z","lastSend":"2026-04-14T17:31:55.3836300Z","lastReceive":"2026-04-14T17:31:55.3836300Z"},"requestSizeInBytes":1630,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:31:55.3523106Z","DurationInMs":15.6997,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"eb31c959-7f6a-4e1d-9956-fcffaaea2fe9","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:31:55.3680103Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"263e63fa-0824-478f-80a2-afd94fe83d6a","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:31:55.3680103Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"cb5ed8cd-03aa-4810-b50f-adcfd9c5f157","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1250357,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"cb5ed8cd-03aa-4810-b50f-adcfd9c5f157","ResponseTimeUtc":"2026-04-14T17:31:55.3992240Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: cb5ed8cd-03aa-4810-b50f-adcfd9c5f157, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.515"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0026000000000000003}]}
2026-04-14 17:37:02 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c69396cb-a07e-4954-bc1d-9568bb3ee056; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c69396cb-a07e-4954-bc1d-9568bb3ee056, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c69396cb-a07e-4954-bc1d-9568bb3ee056; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c69396cb-a07e-4954-bc1d-9568bb3ee056, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:37:02.113Z","duration in milliseconds":40.325900000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:37:02.0194470Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91854,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.051500000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":6.2223000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0024000000000000002},{"name":"Read Collection","duration in milliseconds":6.0428000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:37:02.1131991Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"eb31adcb-fb56-453f-b76f-af035f50df59","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":33.8665,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":33.8255,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:36:06.5448360Z","cpu":2.969,"memory":5225980.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0817,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:36:16.5525366Z","cpu":2.808,"memory":5231840.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.5661,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:36:26.5578851Z","cpu":3.672,"memory":5229816.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0785,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:36:36.5602303Z","cpu":1.250,"memory":5228108.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0352,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:36:46.5704345Z","cpu":0.468,"memory":5241168.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1269,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:36:56.5820653Z","cpu":1.328,"memory":5240484.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.102,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":33.816700000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":33.8121,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":33.7924,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":33.7894,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":32.4962,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:37:02.1288266Z","EndTimeUTC":"2026-04-14T17:37:02.1288266Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:37:02.1444507Z","DurationInMs":15.6241,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c69396cb-a07e-4954-bc1d-9568bb3ee056","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.717","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:37:02 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:37:02.1288266Z", "durationInMs": 0.0078},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:37:02.1288344Z", "durationInMs": 16.6413},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:37:02.1454757Z", "durationInMs": 0.1966},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:37:02.1456723Z", "durationInMs": 1.9613},{"event": "Received", "startTimeUtc": "2026-04-14T17:37:02.1476336Z", "durationInMs": 0.1304},{"event": "Completed", "startTimeUtc": "2026-04-14T17:37:02.1477640Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:37:02.1444507Z","lastSend":"2026-04-14T17:37:02.1444507Z","lastReceive":"2026-04-14T17:37:02.1444507Z"},"requestSizeInBytes":1628,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:37:02.1131991Z","DurationInMs":15.627500000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"311c12ce-f4d0-4bb1-93d5-f5991a8f6502","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:37:02.1288266Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"6dc600cc-e980-4883-9735-1d48fef0842a","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:37:02.1288266Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"c69396cb-a07e-4954-bc1d-9568bb3ee056","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1250037,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c69396cb-a07e-4954-bc1d-9568bb3ee056","ResponseTimeUtc":"2026-04-14T17:37:02.1600798Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c69396cb-a07e-4954-bc1d-9568bb3ee056, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.717"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0025}]}
2026-04-14 17:39:27 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: bb862fac-5d6b-4644-9103-e3f295948cf2; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: bb862fac-5d6b-4644-9103-e3f295948cf2, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: bb862fac-5d6b-4644-9103-e3f295948cf2; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: bb862fac-5d6b-4644-9103-e3f295948cf2, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:39:27.659Z","duration in milliseconds":41.3205,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:39:27.4255518Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91863,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0386},{"name":"Get PkValue From Stream","duration in milliseconds":6.2664,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":6.1245,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:39:27.6599312Z","DurationInMs":15.693200000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"03b140c1-b518-45ae-a3e4-5c831bc55b2d","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":34.8402,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":34.8146,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:38:36.6503192Z","cpu":2.028,"memory":5353140.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1069,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:38:46.6569680Z","cpu":0.859,"memory":5346616.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1049,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:38:56.6676445Z","cpu":2.964,"memory":5292404.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1013,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:39:06.6822205Z","cpu":3.354,"memory":5280804.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.117,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:39:16.6838437Z","cpu":2.500,"memory":5279364.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1088,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:39:26.6860574Z","cpu":0.703,"memory":5278088.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0977,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":34.8083,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":34.803200000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":34.7819,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":34.7798,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":33.93,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:39:27.6756244Z","EndTimeUTC":"2026-04-14T17:39:27.6756244Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:39:27.7068057Z","DurationInMs":31.1813,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"bb862fac-5d6b-4644-9103-e3f295948cf2","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.904","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:39:27 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:39:27.6756244Z", "durationInMs": 0.0075},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:39:27.6756319Z", "durationInMs": 17.2715},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:39:27.6929034Z", "durationInMs": 0.205},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:39:27.6931084Z", "durationInMs": 2.2144},{"event": "Received", "startTimeUtc": "2026-04-14T17:39:27.6953228Z", "durationInMs": 0.123},{"event": "Completed", "startTimeUtc": "2026-04-14T17:39:27.6954458Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:39:27.6911844Z","lastSend":"2026-04-14T17:39:27.6911844Z","lastReceive":"2026-04-14T17:39:27.7068057Z"},"requestSizeInBytes":1630,"requestBodySizeInBytes":1178,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:39:27.6756244Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"ec2a875e-e5cb-47cf-8a6d-18c280989fbe","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:39:27.6756244Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"6cd8eb02-20a8-4f02-93a6-773a16fdc83c","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:39:27.6756244Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"bb862fac-5d6b-4644-9103-e3f295948cf2","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2812539,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"bb862fac-5d6b-4644-9103-e3f295948cf2","ResponseTimeUtc":"2026-04-14T17:39:27.7068057Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: bb862fac-5d6b-4644-9103-e3f295948cf2, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.904"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0026000000000000003}]}
2026-04-14 17:40:32 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 326d171a-8db8-4164-a225-96c335a566b3; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 326d171a-8db8-4164-a225-96c335a566b3, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 326d171a-8db8-4164-a225-96c335a566b3; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 326d171a-8db8-4164-a225-96c335a566b3, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:40:31.974Z","duration in milliseconds":60.8973,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:40:31.7557990Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91874,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0441},{"name":"Get PkValue From Stream","duration in milliseconds":9.708400000000001,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":9.5646,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:40:31.9745515Z","DurationInMs":15.6342,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"2014d1e8-dfb5-4ed4-a9ab-b589857d56df","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":50.9703,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":50.936800000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:39:36.6881056Z","cpu":2.031,"memory":5273636.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1228,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:39:46.6922006Z","cpu":1.875,"memory":5274508.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1023,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:39:56.7071268Z","cpu":0.234,"memory":5272292.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1135,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:40:06.7222112Z","cpu":2.340,"memory":5259380.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1226,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:40:16.7246611Z","cpu":3.672,"memory":5214008.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0912,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:40:26.7392451Z","cpu":1.638,"memory":5211896.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0529,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":50.9288,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":50.9249,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":50.905,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":50.902100000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":49.940400000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:40:32.0058620Z","EndTimeUTC":"2026-04-14T17:40:32.0058620Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:40:32.0371309Z","DurationInMs":31.268900000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"326d171a-8db8-4164-a225-96c335a566b3","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.77","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:40:32 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:40:32.0058620Z", "durationInMs": 0.0082},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:40:32.0058702Z", "durationInMs": 32.663},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:40:32.0385332Z", "durationInMs": 0.2056},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:40:32.0387388Z", "durationInMs": 2.667},{"event": "Received", "startTimeUtc": "2026-04-14T17:40:32.0414058Z", "durationInMs": 0.1472},{"event": "Completed", "startTimeUtc": "2026-04-14T17:40:32.0415530Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:40:32.0371309Z","lastSend":"2026-04-14T17:40:32.0371309Z","lastReceive":"2026-04-14T17:40:32.0371309Z"},"requestSizeInBytes":1627,"requestBodySizeInBytes":1179,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:40:31.9901857Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"73fbcf0c-dfd4-4170-a84b-7a80ec1e3793","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:40:31.9901857Z","DurationInMs":15.676300000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"9a43a70e-f81b-424e-86e2-5fbeaa6907f3","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:40:32.0058620Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"326d171a-8db8-4164-a225-96c335a566b3","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.28133189999999997,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"326d171a-8db8-4164-a225-96c335a566b3","ResponseTimeUtc":"2026-04-14T17:40:32.0371309Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 326d171a-8db8-4164-a225-96c335a566b3, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.77"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 17:41:00 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 216d230c-a9f9-4fff-a543-380e3e2253d9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 216d230c-a9f9-4fff-a543-380e3e2253d9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 216d230c-a9f9-4fff-a543-380e3e2253d9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 216d230c-a9f9-4fff-a543-380e3e2253d9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:41:00.423Z","duration in milliseconds":44.045100000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:41:00.3454135Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91877,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.051800000000000006},{"name":"Get PkValue From Stream","duration in milliseconds":6.4037000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003},{"name":"Read Collection","duration in milliseconds":6.231,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:41:00.4239935Z","DurationInMs":15.6872,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"b3722479-a4ef-4bd1-b477-e9c3e4162315","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.401700000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.3611,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:40:06.7222112Z","cpu":2.340,"memory":5259380.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1226,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:40:16.7246611Z","cpu":3.672,"memory":5214008.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0912,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:40:26.7392451Z","cpu":1.638,"memory":5211896.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0529,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:40:36.7411817Z","cpu":2.344,"memory":5211856.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1104,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:40:46.7565968Z","cpu":0.780,"memory":5211556.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1069,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:40:56.7672399Z","cpu":2.028,"memory":5211768.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1096,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.3534,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.3487,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.3252,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.3224,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.441700000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:41:00.4396807Z","EndTimeUTC":"2026-04-14T17:41:00.4552369Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:41:00.4708629Z","DurationInMs":15.626000000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"216d230c-a9f9-4fff-a543-380e3e2253d9","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.665","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:41:00 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:41:00.4552369Z", "durationInMs": 0.0088},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:41:00.4552457Z", "durationInMs": 18.0199},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:41:00.4732656Z", "durationInMs": 0.2254},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:41:00.4734910Z", "durationInMs": 1.8808},{"event": "Received", "startTimeUtc": "2026-04-14T17:41:00.4753718Z", "durationInMs": 0.1546},{"event": "Completed", "startTimeUtc": "2026-04-14T17:41:00.4755264Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:41:00.4708629Z","lastSend":"2026-04-14T17:41:00.4708629Z","lastReceive":"2026-04-14T17:41:00.4708629Z"},"requestSizeInBytes":1635,"requestBodySizeInBytes":1187,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:41:00.4396807Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"1ce505e3-2583-488d-af7d-facc4dffd0c0","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:41:00.4396807Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"81af4366-acb2-4595-b7fb-c1d9be35dc75","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:41:00.4396807Z","DurationInMs":15.5562,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"216d230c-a9f9-4fff-a543-380e3e2253d9","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1254494,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"216d230c-a9f9-4fff-a543-380e3e2253d9","ResponseTimeUtc":"2026-04-14T17:41:00.4708629Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 216d230c-a9f9-4fff-a543-380e3e2253d9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.665"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 17:41:26 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b2cf313c-df25-4ac0-a70c-0622d9840237; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b2cf313c-df25-4ac0-a70c-0622d9840237, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b2cf313c-df25-4ac0-a70c-0622d9840237; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b2cf313c-df25-4ac0-a70c-0622d9840237, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:41:26.783Z","duration in milliseconds":41.897800000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:41:26.6896273Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91883,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.048400000000000006},{"name":"Get PkValue From Stream","duration in milliseconds":6.4622,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0033},{"name":"Read Collection","duration in milliseconds":6.2918,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:41:26.7834997Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"94b1c781-a8ef-4cce-b3c6-e912032e5abb","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.1935,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.1548,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:40:36.7411817Z","cpu":2.344,"memory":5211856.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1104,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:40:46.7565968Z","cpu":0.780,"memory":5211556.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1069,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:40:56.7672399Z","cpu":2.028,"memory":5211768.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1096,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:06.7826034Z","cpu":4.056,"memory":5202228.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1111,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:16.7946207Z","cpu":3.276,"memory":5196704.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1021,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:26.7990087Z","cpu":9.297,"memory":4869032.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0448,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.144200000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.1374,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.1151,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.112300000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.2003,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:41:26.7990087Z","EndTimeUTC":"2026-04-14T17:41:26.7990087Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:41:26.8146286Z","DurationInMs":15.619900000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"b2cf313c-df25-4ac0-a70c-0622d9840237","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.575","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:41:26 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:41:26.7990087Z", "durationInMs": 0.0102},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:41:26.7990189Z", "durationInMs": 17.4},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:41:26.8164189Z", "durationInMs": 0.1942},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:41:26.8166131Z", "durationInMs": 1.9603},{"event": "Received", "startTimeUtc": "2026-04-14T17:41:26.8185734Z", "durationInMs": 0.1491},{"event": "Completed", "startTimeUtc": "2026-04-14T17:41:26.8187225Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:41:26.8146286Z","lastSend":"2026-04-14T17:41:26.8146286Z","lastReceive":"2026-04-14T17:41:26.8146286Z"},"requestSizeInBytes":1628,"requestBodySizeInBytes":1178,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:41:26.7834997Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0d77ba9c-b95f-4bb3-a059-24441cc6cd3d","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:41:26.7834997Z","DurationInMs":15.509,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"7d116ba5-17e2-4e06-95fa-ad67c74f8a1b","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:41:26.7990087Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"b2cf313c-df25-4ac0-a70c-0622d9840237","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.12500129999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"b2cf313c-df25-4ac0-a70c-0622d9840237","ResponseTimeUtc":"2026-04-14T17:41:26.8146286Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: b2cf313c-df25-4ac0-a70c-0622d9840237, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.575"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 17:41:48 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 887a4a54-11d1-48b1-b992-8f65de1f61a1; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 887a4a54-11d1-48b1-b992-8f65de1f61a1, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 887a4a54-11d1-48b1-b992-8f65de1f61a1; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 887a4a54-11d1-48b1-b992-8f65de1f61a1, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:41:48.919Z","duration in milliseconds":51.3003,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:41:48.8415935Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91886,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0505},{"name":"Get PkValue From Stream","duration in milliseconds":6.6485,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0024000000000000002},{"name":"Read Collection","duration in milliseconds":6.479,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:41:48.9197216Z","DurationInMs":15.623600000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"d1776f29-f7df-442d-8898-5c15558f18c1","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":44.4213,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":44.386500000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:40:56.7672399Z","cpu":2.028,"memory":5211768.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1096,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:06.7826034Z","cpu":4.056,"memory":5202228.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1111,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:16.7946207Z","cpu":3.276,"memory":5196704.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1021,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:26.7990087Z","cpu":9.297,"memory":4869032.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0448,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1},{"dateUtc":"2026-04-14T17:41:36.8032078Z","cpu":4.297,"memory":4999948.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0744,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:46.8103907Z","cpu":4.056,"memory":5153024.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2124,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":44.3784,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":44.3744,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":44.3552,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":44.3525,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":43.459500000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:41:48.9353452Z","EndTimeUTC":"2026-04-14T17:41:48.9509687Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:41:48.9665957Z","DurationInMs":15.627,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"887a4a54-11d1-48b1-b992-8f65de1f61a1","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.835","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:41:48 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:41:48.9509687Z", "durationInMs": 0.0103},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:41:48.9509790Z", "durationInMs": 25.2458},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:41:48.9762248Z", "durationInMs": 0.2066},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:41:48.9764314Z", "durationInMs": 2.3675},{"event": "Received", "startTimeUtc": "2026-04-14T17:41:48.9787989Z", "durationInMs": 0.1351},{"event": "Completed", "startTimeUtc": "2026-04-14T17:41:48.9789340Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:41:48.9665957Z","lastSend":"2026-04-14T17:41:48.9665957Z","lastReceive":"2026-04-14T17:41:48.9665957Z"},"requestSizeInBytes":1635,"requestBodySizeInBytes":1187,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:41:48.9353452Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"f38a093e-b947-45a0-a0b4-8a355ace5aa2","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:41:48.9353452Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"04214248-fa5d-42c1-94f1-6e6f10e6a543","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:41:48.9353452Z","DurationInMs":15.6235,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"887a4a54-11d1-48b1-b992-8f65de1f61a1","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1250022,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"887a4a54-11d1-48b1-b992-8f65de1f61a1","ResponseTimeUtc":"2026-04-14T17:41:48.9665957Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 887a4a54-11d1-48b1-b992-8f65de1f61a1, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.835"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0015}]}
2026-04-14 17:42:04 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c501b902-63ff-423a-b1e1-02c3733b43f9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c501b902-63ff-423a-b1e1-02c3733b43f9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c501b902-63ff-423a-b1e1-02c3733b43f9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c501b902-63ff-423a-b1e1-02c3733b43f9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:42:04.212Z","duration in milliseconds":42.7042,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:42:04.1347724Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91888,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0507},{"name":"Get PkValue From Stream","duration in milliseconds":6.767,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0026000000000000003},{"name":"Read Collection","duration in milliseconds":6.5827,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:42:04.2285245Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"152b105c-8b28-497a-97b0-467cd7baeaa4","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.6633,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.6225,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:41:06.7826034Z","cpu":4.056,"memory":5202228.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1111,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:16.7946207Z","cpu":3.276,"memory":5196704.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1021,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:26.7990087Z","cpu":9.297,"memory":4869032.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0448,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1},{"dateUtc":"2026-04-14T17:41:36.8032078Z","cpu":4.297,"memory":4999948.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0744,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:46.8103907Z","cpu":4.056,"memory":5153024.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2124,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:56.8243520Z","cpu":4.290,"memory":5202068.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0871,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.614000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.6107,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.5889,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.5863,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.2003,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:42:04.2441463Z","EndTimeUTC":"2026-04-14T17:42:04.2441463Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:42:04.2597827Z","DurationInMs":15.6364,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c501b902-63ff-423a-b1e1-02c3733b43f9","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.661","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:42:04 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:42:04.2441463Z", "durationInMs": 0.01},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:42:04.2441563Z", "durationInMs": 17.6292},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:42:04.2617855Z", "durationInMs": 0.2143},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:42:04.2619998Z", "durationInMs": 2.0712},{"event": "Received", "startTimeUtc": "2026-04-14T17:42:04.2640710Z", "durationInMs": 0.103},{"event": "Completed", "startTimeUtc": "2026-04-14T17:42:04.2641740Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:42:04.2597827Z","lastSend":"2026-04-14T17:42:04.2597827Z","lastReceive":"2026-04-14T17:42:04.2597827Z"},"requestSizeInBytes":1639,"requestBodySizeInBytes":1187,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:42:04.2285245Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"4bd63bac-7932-4177-9c0b-11af002b1f57","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:42:04.2285245Z","DurationInMs":15.6218,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"219d52fa-1442-4486-8a47-f7996a3106eb","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:42:04.2441463Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"c501b902-63ff-423a-b1e1-02c3733b43f9","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1250103,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c501b902-63ff-423a-b1e1-02c3733b43f9","ResponseTimeUtc":"2026-04-14T17:42:04.2597827Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c501b902-63ff-423a-b1e1-02c3733b43f9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.661"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0015}]}
2026-04-14 17:42:17 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 17a7c5b1-7929-4de3-9178-a07dad0ba9ea; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 17a7c5b1-7929-4de3-9178-a07dad0ba9ea, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 17a7c5b1-7929-4de3-9178-a07dad0ba9ea; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 17a7c5b1-7929-4de3-9178-a07dad0ba9ea, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:42:16.976Z","duration in milliseconds":43.7442,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:42:16.7732541Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91891,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.048},{"name":"Get PkValue From Stream","duration in milliseconds":7.0218000000000007,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0013000000000000002},{"name":"Read Collection","duration in milliseconds":6.8610000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:42:16.9920011Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"e1f44e59-80bb-424f-bffd-d4d28d2a819a","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.493300000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.461400000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:41:26.7990087Z","cpu":9.297,"memory":4869032.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0448,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1},{"dateUtc":"2026-04-14T17:41:36.8032078Z","cpu":4.297,"memory":4999948.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0744,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:46.8103907Z","cpu":4.056,"memory":5153024.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2124,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:56.8243520Z","cpu":4.290,"memory":5202068.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0871,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:06.8299022Z","cpu":3.125,"memory":5199368.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1291,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:16.8357487Z","cpu":7.734,"memory":5200488.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1185,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.4536,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.449600000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.428200000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.4259,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.571000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:42:17.0076868Z","EndTimeUTC":"2026-04-14T17:42:17.0076868Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:42:17.0232516Z","DurationInMs":15.5648,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"17a7c5b1-7929-4de3-9178-a07dad0ba9ea","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.771","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:42:17 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:42:17.0076868Z", "durationInMs": 0.0088},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:42:17.0076956Z", "durationInMs": 17.3645},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:42:17.0250601Z", "durationInMs": 0.1968},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:42:17.0252569Z", "durationInMs": 1.9847},{"event": "Received", "startTimeUtc": "2026-04-14T17:42:17.0272416Z", "durationInMs": 0.1358},{"event": "Completed", "startTimeUtc": "2026-04-14T17:42:17.0273774Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:42:17.0232516Z","lastSend":"2026-04-14T17:42:17.0232516Z","lastReceive":"2026-04-14T17:42:17.0232516Z"},"requestSizeInBytes":1630,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:42:16.9920011Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"80884f96-4b2e-4ffc-8263-9d0f07c18591","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:42:16.9920011Z","DurationInMs":15.6857,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"99b3e1f5-2ce5-47cd-b69a-ee9ac959b21c","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:42:17.0076868Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"17a7c5b1-7929-4de3-9178-a07dad0ba9ea","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.24999749999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"17a7c5b1-7929-4de3-9178-a07dad0ba9ea","ResponseTimeUtc":"2026-04-14T17:42:17.0232516Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 17a7c5b1-7929-4de3-9178-a07dad0ba9ea, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.771"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 17:42:32 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b2167021-1294-414e-8973-6db57b7c3394; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b2167021-1294-414e-8973-6db57b7c3394, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b2167021-1294-414e-8973-6db57b7c3394; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b2167021-1294-414e-8973-6db57b7c3394, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:42:32.710Z","duration in milliseconds":50.400200000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:42:32.6481608Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91897,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0511},{"name":"Get PkValue From Stream","duration in milliseconds":8.0542,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0019},{"name":"Read Collection","duration in milliseconds":7.8770000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:42:32.7262855Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"2c337e63-47da-471a-83f1-aafe9767f04d","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":42.1053,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":42.065200000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:41:36.8032078Z","cpu":4.297,"memory":4999948.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0744,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:46.8103907Z","cpu":4.056,"memory":5153024.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2124,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:56.8243520Z","cpu":4.290,"memory":5202068.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0871,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:06.8299022Z","cpu":3.125,"memory":5199368.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1291,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:16.8357487Z","cpu":7.734,"memory":5200488.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1185,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:26.8400984Z","cpu":4.680,"memory":5216068.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0312,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":42.057,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":42.0529,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":42.0309,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":42.0283,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":41.1184,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:42:32.7419641Z","EndTimeUTC":"2026-04-14T17:42:32.7419641Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:42:32.7741762Z","DurationInMs":32.2121,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"b2167021-1294-414e-8973-6db57b7c3394","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.7","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:42:32 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:42:32.7419641Z", "durationInMs": 0.0075},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:42:32.7419716Z", "durationInMs": 18.7249},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:42:32.7606965Z", "durationInMs": 0.2179},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:42:32.7609144Z", "durationInMs": 6.1033},{"event": "Received", "startTimeUtc": "2026-04-14T17:42:32.7670177Z", "durationInMs": 0.1469},{"event": "Completed", "startTimeUtc": "2026-04-14T17:42:32.7671646Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:42:32.7575374Z","lastSend":"2026-04-14T17:42:32.7575374Z","lastReceive":"2026-04-14T17:42:32.7575374Z"},"requestSizeInBytes":1635,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:42:32.7262855Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"b57fddce-2ce2-4c17-819a-a1f90e306a18","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:42:32.7262855Z","DurationInMs":15.678600000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"d592597c-4886-44c9-aba6-4a41542aeb7b","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:42:32.7419641Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"b2167021-1294-414e-8973-6db57b7c3394","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1260154,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"b2167021-1294-414e-8973-6db57b7c3394","ResponseTimeUtc":"2026-04-14T17:42:32.7741762Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: b2167021-1294-414e-8973-6db57b7c3394, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.7"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 17:42:43 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c811a5a9-b167-44ba-9f3d-6a9726dcb654; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c811a5a9-b167-44ba-9f3d-6a9726dcb654, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c811a5a9-b167-44ba-9f3d-6a9726dcb654; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c811a5a9-b167-44ba-9f3d-6a9726dcb654, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:42:43.816Z","duration in milliseconds":45.605900000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:42:43.7224112Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91900,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.049800000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":6.2534,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.077,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:42:43.8161584Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"3b0d14c3-2498-41d3-870b-47d6a68c39e3","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":39.045700000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":38.9998,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:41:46.8103907Z","cpu":4.056,"memory":5153024.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2124,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:41:56.8243520Z","cpu":4.290,"memory":5202068.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0871,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:06.8299022Z","cpu":3.125,"memory":5199368.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1291,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:16.8357487Z","cpu":7.734,"memory":5200488.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1185,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:26.8400984Z","cpu":4.680,"memory":5216068.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0312,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:36.8458488Z","cpu":13.516,"memory":5144588.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0626,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":38.988,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":38.9823,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":38.9575,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":38.9528,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":37.763000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:42:43.8317837Z","EndTimeUTC":"2026-04-14T17:42:43.8317837Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:42:43.8474123Z","DurationInMs":15.6286,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c811a5a9-b167-44ba-9f3d-6a9726dcb654","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.6","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:42:43 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:42:43.8317837Z", "durationInMs": 0.0078},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:42:43.8317915Z", "durationInMs": 21.0952},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:42:43.8528867Z", "durationInMs": 0.2527},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:42:43.8531394Z", "durationInMs": 1.9164},{"event": "Received", "startTimeUtc": "2026-04-14T17:42:43.8550558Z", "durationInMs": 0.1576},{"event": "Completed", "startTimeUtc": "2026-04-14T17:42:43.8552134Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:42:43.8474123Z","lastSend":"2026-04-14T17:42:43.8474123Z","lastReceive":"2026-04-14T17:42:43.8474123Z"},"requestSizeInBytes":1633,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:42:43.8161584Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"8da9159c-e7b6-4a1e-a1bf-77d8ad129dc6","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:42:43.8161584Z","DurationInMs":15.625300000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"94687539-873c-4f1e-a794-e94fac8c5273","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:42:43.8317837Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"c811a5a9-b167-44ba-9f3d-6a9726dcb654","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1250011,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c811a5a9-b167-44ba-9f3d-6a9726dcb654","ResponseTimeUtc":"2026-04-14T17:42:43.8630348Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c811a5a9-b167-44ba-9f3d-6a9726dcb654, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.6"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 17:43:04 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5c6c41ef-5703-4e65-b22e-d4fa99548b86; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5c6c41ef-5703-4e65-b22e-d4fa99548b86, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5c6c41ef-5703-4e65-b22e-d4fa99548b86; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5c6c41ef-5703-4e65-b22e-d4fa99548b86, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:43:04.621Z","duration in milliseconds":49.2205,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:43:04.5436820Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91908,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.060200000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":6.1532,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0024000000000000002},{"name":"Read Collection","duration in milliseconds":5.9687,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:43:04.6218222Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"1e696798-c5ab-465d-870a-9b06d611e367","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":42.7748,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":42.7318,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:42:06.8299022Z","cpu":3.125,"memory":5199368.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1291,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:16.8357487Z","cpu":7.734,"memory":5200488.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1185,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:26.8400984Z","cpu":4.680,"memory":5216068.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0312,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:36.8458488Z","cpu":13.516,"memory":5144588.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0626,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:46.8481630Z","cpu":3.125,"memory":5140524.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0993,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:56.8557574Z","cpu":6.084,"memory":5158940.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0989,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":42.7216,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":42.7165,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":42.689,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":42.6863,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":41.6417,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:43:04.6375100Z","EndTimeUTC":"2026-04-14T17:43:04.6375100Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:43:04.6686836Z","DurationInMs":31.1736,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"5c6c41ef-5703-4e65-b22e-d4fa99548b86","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.55","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:43:04 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:43:04.6375100Z", "durationInMs": 0.0085},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:43:04.6375185Z", "durationInMs": 19.5965},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:43:04.6571150Z", "durationInMs": 0.1953},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:43:04.6573103Z", "durationInMs": 2.5037},{"event": "Received", "startTimeUtc": "2026-04-14T17:43:04.6598140Z", "durationInMs": 0.1245},{"event": "Completed", "startTimeUtc": "2026-04-14T17:43:04.6599385Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:43:04.6531331Z","lastSend":"2026-04-14T17:43:04.6531331Z","lastReceive":"2026-04-14T17:43:04.6531331Z"},"requestSizeInBytes":1633,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:43:04.6218222Z","DurationInMs":15.687800000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"06e213db-ddb9-435e-9a4d-7fddff83ae9e","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:43:04.6375100Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"091f0f5b-1bd3-4629-9ef4-1c4606a22d24","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:43:04.6375100Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"5c6c41ef-5703-4e65-b22e-d4fa99548b86","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1250016,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"5c6c41ef-5703-4e65-b22e-d4fa99548b86","ResponseTimeUtc":"2026-04-14T17:43:04.6686836Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 5c6c41ef-5703-4e65-b22e-d4fa99548b86, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.55"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0033}]}
2026-04-14 17:43:07 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 41608912-b2d8-4c83-9308-1022e46c1a2e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 41608912-b2d8-4c83-9308-1022e46c1a2e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 41608912-b2d8-4c83-9308-1022e46c1a2e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 41608912-b2d8-4c83-9308-1022e46c1a2e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:43:07.705Z","duration in milliseconds":47.5383,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:43:07.5962330Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91909,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0356},{"name":"Get PkValue From Stream","duration in milliseconds":7.626,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.001},{"name":"Read Collection","duration in milliseconds":7.4039,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:43:07.7056052Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"fd04a1fb-527f-4162-9efd-db5bdd9f7bb0","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":39.6706,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":39.6328,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:42:16.8357487Z","cpu":7.734,"memory":5200488.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1185,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:26.8400984Z","cpu":4.680,"memory":5216068.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0312,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:36.8458488Z","cpu":13.516,"memory":5144588.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0626,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:46.8481630Z","cpu":3.125,"memory":5140524.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0993,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:42:56.8557574Z","cpu":6.084,"memory":5158940.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0989,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:43:06.8618400Z","cpu":8.906,"memory":5293004.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1006,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":39.626000000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":39.6206,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":39.5983,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":39.595,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":38.3849,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:43:07.7212355Z","EndTimeUTC":"2026-04-14T17:43:07.7212355Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:43:07.7524819Z","DurationInMs":31.2464,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"41608912-b2d8-4c83-9308-1022e46c1a2e","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.662","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:43:07 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:43:07.7212355Z", "durationInMs": 0.0097},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:43:07.7212452Z", "durationInMs": 21.8354},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:43:07.7430806Z", "durationInMs": 0.1784},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:43:07.7432590Z", "durationInMs": 1.8832},{"event": "Received", "startTimeUtc": "2026-04-14T17:43:07.7451422Z", "durationInMs": 0.1288},{"event": "Completed", "startTimeUtc": "2026-04-14T17:43:07.7452710Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:43:07.7368689Z","lastSend":"2026-04-14T17:43:07.7368689Z","lastReceive":"2026-04-14T17:43:07.7368689Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:43:07.7056052Z","DurationInMs":15.6303,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"d5b04ab7-8369-4302-87d8-59bf90ec5446","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:43:07.7212355Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"12ccc8f8-18d0-4584-876c-626ce3d66ecb","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:43:07.7212355Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"41608912-b2d8-4c83-9308-1022e46c1a2e","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718756,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"41608912-b2d8-4c83-9308-1022e46c1a2e","ResponseTimeUtc":"2026-04-14T17:43:07.7524819Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 41608912-b2d8-4c83-9308-1022e46c1a2e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.662"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 17:44:04 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: f5d8237f-c4c2-4605-ac08-7001d78b9b54; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: f5d8237f-c4c2-4605-ac08-7001d78b9b54, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: f5d8237f-c4c2-4605-ac08-7001d78b9b54; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: f5d8237f-c4c2-4605-ac08-7001d78b9b54, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:44:04.243Z","duration in milliseconds":43.8369,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:44:04.1656505Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91916,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0499},{"name":"Get PkValue From Stream","duration in milliseconds":6.8894,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014},{"name":"Read Collection","duration in milliseconds":6.7224,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:44:04.2438620Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"7a1bbe34-8d30-481c-b4a6-eded847c5a85","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.7156,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.6796,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:43:06.8618400Z","cpu":8.906,"memory":5293004.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1006,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:43:16.8720191Z","cpu":10.140,"memory":5284932.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0942,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:43:26.8806861Z","cpu":2.344,"memory":5238900.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1171,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:43:36.8865834Z","cpu":0.391,"memory":5235216.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1093,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:43:46.8904689Z","cpu":1.404,"memory":5240092.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0955,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:43:56.9031366Z","cpu":0.703,"memory":5229376.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1113,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.6724,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.6674,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.6503,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.6475,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.792100000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:44:04.2594051Z","EndTimeUTC":"2026-04-14T17:44:04.2594051Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:44:04.2750285Z","DurationInMs":15.6234,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"f5d8237f-c4c2-4605-ac08-7001d78b9b54","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.638","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:44:04 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:44:04.2594051Z", "durationInMs": 0.0098},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:44:04.2594149Z", "durationInMs": 18.5692},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:44:04.2779841Z", "durationInMs": 0.1921},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:44:04.2781762Z", "durationInMs": 2.2097},{"event": "Received", "startTimeUtc": "2026-04-14T17:44:04.2803859Z", "durationInMs": 0.1335},{"event": "Completed", "startTimeUtc": "2026-04-14T17:44:04.2805194Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:44:04.2750285Z","lastSend":"2026-04-14T17:44:04.2750285Z","lastReceive":"2026-04-14T17:44:04.2750285Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1186,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:44:04.2438620Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"525becfd-9227-4252-afff-4228ffc86aae","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:44:04.2438620Z","DurationInMs":15.5431,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"ee6bb480-4a47-48d3-ad56-a207334f265a","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:44:04.2594051Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"f5d8237f-c4c2-4605-ac08-7001d78b9b54","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.10937799999999999,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"f5d8237f-c4c2-4605-ac08-7001d78b9b54","ResponseTimeUtc":"2026-04-14T17:44:04.2750285Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: f5d8237f-c4c2-4605-ac08-7001d78b9b54, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.638"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001}]}
2026-04-14 17:44:33 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 785e8523-a5ae-4cbe-8054-b0fb8d41cdcb; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 785e8523-a5ae-4cbe-8054-b0fb8d41cdcb, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 785e8523-a5ae-4cbe-8054-b0fb8d41cdcb; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 785e8523-a5ae-4cbe-8054-b0fb8d41cdcb, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:44:33.086Z","duration in milliseconds":61.991200000000006,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:44:32.8516662Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91919,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0485},{"name":"Get PkValue From Stream","duration in milliseconds":7.0672000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0035},{"name":"Read Collection","duration in milliseconds":6.9035,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:44:33.0860436Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"d381a612-00ab-4cb7-8adc-868cfe737314","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":54.6961,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":54.6651,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:43:36.8865834Z","cpu":0.391,"memory":5235216.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1093,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:43:46.8904689Z","cpu":1.404,"memory":5240092.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0955,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:43:56.9031366Z","cpu":0.703,"memory":5229376.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1113,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:44:06.9075887Z","cpu":2.184,"memory":5225224.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1125,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:44:16.9173879Z","cpu":4.297,"memory":5221724.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2701,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:44:26.9207437Z","cpu":4.056,"memory":5238988.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1236,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":54.6567,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":54.6514,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":54.629400000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":54.625,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":53.7639,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:44:33.1016694Z","EndTimeUTC":"2026-04-14T17:44:33.1016694Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:44:33.1485425Z","DurationInMs":46.8731,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"785e8523-a5ae-4cbe-8054-b0fb8d41cdcb","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149651,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149651,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149651,"SessionToken":"-1#26149651","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.676","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:44:33 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:44:33.1016694Z", "durationInMs": 0.0093},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:44:33.1016787Z", "durationInMs": 35.9496},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:44:33.1376283Z", "durationInMs": 0.2347},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:44:33.1378630Z", "durationInMs": 1.7762},{"event": "Received", "startTimeUtc": "2026-04-14T17:44:33.1396392Z", "durationInMs": 0.146},{"event": "Completed", "startTimeUtc": "2026-04-14T17:44:33.1397852Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:44:33.1485425Z","lastSend":"2026-04-14T17:44:33.1485425Z","lastReceive":"2026-04-14T17:44:33.1485425Z"},"requestSizeInBytes":1628,"requestBodySizeInBytes":1178,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:44:33.0860436Z","DurationInMs":15.6258,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"3a2c1da3-b8dc-45d4-b4ed-7620df2f1e9e","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:44:33.1016694Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"653f5fd8-ddbd-4a83-9cec-78a1bcaed22f","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:44:33.1016694Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"785e8523-a5ae-4cbe-8054-b0fb8d41cdcb","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2968763,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"785e8523-a5ae-4cbe-8054-b0fb8d41cdcb","ResponseTimeUtc":"2026-04-14T17:44:33.1485425Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 785e8523-a5ae-4cbe-8054-b0fb8d41cdcb, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149651","BELatencyInMs":"0.676"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0015}]}
2026-04-14 17:48:15 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 42359a5a-dbda-41d3-8cdd-18ec6d11459d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 42359a5a-dbda-41d3-8cdd-18ec6d11459d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 42359a5a-dbda-41d3-8cdd-18ec6d11459d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 42359a5a-dbda-41d3-8cdd-18ec6d11459d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:48:15.933Z","duration in milliseconds":48.010000000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:48:15.8553177Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91943,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.047400000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":6.4492,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001},{"name":"Read Collection","duration in milliseconds":6.2855,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:48:15.9334449Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"d49a05b3-6dc1-45ed-a123-4b38efb0868e","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":41.335300000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":41.297200000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:47:17.0847370Z","cpu":1.716,"memory":5165396.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0688,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:47:27.0870543Z","cpu":2.422,"memory":5165156.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0829,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:47:37.0937324Z","cpu":0.781,"memory":5164916.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0963,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:47:47.0976641Z","cpu":2.262,"memory":5170908.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0953,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:47:57.1091477Z","cpu":1.172,"memory":5180788.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3159,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:48:07.1232419Z","cpu":2.028,"memory":5274056.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.084,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":41.2879,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":41.283500000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":41.263200000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":41.260400000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":40.289,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:48:15.9491759Z","EndTimeUTC":"2026-04-14T17:48:15.9491759Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:48:15.9803219Z","DurationInMs":31.146,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"42359a5a-dbda-41d3-8cdd-18ec6d11459d","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149701,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149701,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149701,"SessionToken":"-1#26149701","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.731","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:48:15 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:48:15.9491759Z", "durationInMs": 0.0073},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:48:15.9491832Z", "durationInMs": 20.7413},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:48:15.9699245Z", "durationInMs": 0.2123},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:48:15.9701368Z", "durationInMs": 2.2145},{"event": "Received", "startTimeUtc": "2026-04-14T17:48:15.9723513Z", "durationInMs": 0.2904},{"event": "Completed", "startTimeUtc": "2026-04-14T17:48:15.9726417Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:48:15.9646973Z","lastSend":"2026-04-14T17:48:15.9646973Z","lastReceive":"2026-04-14T17:48:15.9803219Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:48:15.9334449Z","DurationInMs":15.731,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"47404402-66d8-4e21-bf3c-2812ca51ce7e","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:48:15.9491759Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"31845276-1913-49e2-bcd9-a1d0adb2a4e8","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:48:15.9491759Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"42359a5a-dbda-41d3-8cdd-18ec6d11459d","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.12500419999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"42359a5a-dbda-41d3-8cdd-18ec6d11459d","ResponseTimeUtc":"2026-04-14T17:48:15.9803219Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 42359a5a-dbda-41d3-8cdd-18ec6d11459d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149701","BELatencyInMs":"0.731"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0023}]}
2026-04-14 17:52:07 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 1480e12a-86e7-42b1-a996-60eb2c6cc946; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 1480e12a-86e7-42b1-a996-60eb2c6cc946, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 1480e12a-86e7-42b1-a996-60eb2c6cc946; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 1480e12a-86e7-42b1-a996-60eb2c6cc946, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:52:07.653Z","duration in milliseconds":42.9957,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:52:07.1064941Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91962,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0453},{"name":"Get PkValue From Stream","duration in milliseconds":6.6022000000000007,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0033},{"name":"Read Collection","duration in milliseconds":6.4523,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:52:07.6535311Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"b8b4a470-5910-42f9-8a8f-f65b8e22d23e","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.17,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.1368,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:51:17.2870554Z","cpu":1.170,"memory":5230812.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0908,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:51:27.2993327Z","cpu":0.547,"memory":5234528.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1273,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:51:37.3038621Z","cpu":0.936,"memory":5234424.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0947,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:51:47.3132121Z","cpu":3.359,"memory":5235040.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1479,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:51:57.3237700Z","cpu":0.156,"memory":5235988.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1395,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:52:07.3253044Z","cpu":6.875,"memory":5219864.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1313,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.1293,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.1259,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.1062,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.1032,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.1591,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:52:07.6689977Z","EndTimeUTC":"2026-04-14T17:52:07.6689977Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:52:07.6846256Z","DurationInMs":15.6279,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"1480e12a-86e7-42b1-a996-60eb2c6cc946","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149711,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149711,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149711,"SessionToken":"-1#26149711","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.726","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:52:07 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:52:07.6689977Z", "durationInMs": 0.0087},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:52:07.6690064Z", "durationInMs": 18.9028},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:52:07.6879092Z", "durationInMs": 0.2735},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:52:07.6881827Z", "durationInMs": 1.9496},{"event": "Received", "startTimeUtc": "2026-04-14T17:52:07.6901323Z", "durationInMs": 0.1442},{"event": "Completed", "startTimeUtc": "2026-04-14T17:52:07.6902765Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:52:07.6846256Z","lastSend":"2026-04-14T17:52:07.6846256Z","lastReceive":"2026-04-14T17:52:07.6846256Z"},"requestSizeInBytes":1596,"requestBodySizeInBytes":1146,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:52:07.6535311Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"22a0ddc9-628c-4b4c-bd81-16279af1337f","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:52:07.6535311Z","DurationInMs":15.466600000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"04f12821-142c-4193-a8c0-7bbbdf4655f2","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:52:07.6689977Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"1480e12a-86e7-42b1-a996-60eb2c6cc946","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.59374739999999993,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"1480e12a-86e7-42b1-a996-60eb2c6cc946","ResponseTimeUtc":"2026-04-14T17:52:07.6846256Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 1480e12a-86e7-42b1-a996-60eb2c6cc946, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149711","BELatencyInMs":"0.726"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0023}]}
2026-04-14 17:53:24 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: aaae6ead-1604-48b4-94b8-8b8c63970d5b; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: aaae6ead-1604-48b4-94b8-8b8c63970d5b, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: aaae6ead-1604-48b4-94b8-8b8c63970d5b; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: aaae6ead-1604-48b4-94b8-8b8c63970d5b, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:53:24.003Z","duration in milliseconds":54.1147,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:53:23.7069506Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":91985,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.054200000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":8.6053,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0041},{"name":"Read Collection","duration in milliseconds":8.4375,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:53:24.0038418Z","DurationInMs":15.624600000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"32ee3a7f-ea2f-4a2d-ad78-d38f77ea052b","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":45.1575,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":45.1177,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:52:27.3387982Z","cpu":1.250,"memory":5244072.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0996,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:52:37.3524904Z","cpu":0.780,"memory":5242264.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0854,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:52:47.3603658Z","cpu":1.172,"memory":5239412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1113,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:52:57.3700645Z","cpu":3.588,"memory":5238324.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0671,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:53:07.3852187Z","cpu":1.404,"memory":5345128.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1378,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:53:17.3944705Z","cpu":4.766,"memory":5339052.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1512,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":45.1069,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":45.0707,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":45.0493,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":45.0462,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":44.037400000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:53:24.0194664Z","EndTimeUTC":"2026-04-14T17:53:24.0351470Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:53:24.0507169Z","DurationInMs":15.5699,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"aaae6ead-1604-48b4-94b8-8b8c63970d5b","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149711,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149711,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149711,"SessionToken":"-1#26149711","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.598","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:53:24 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:53:24.0351470Z", "durationInMs": 0.0077},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:53:24.0351547Z", "durationInMs": 24.9643},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:53:24.0601190Z", "durationInMs": 0.2219},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:53:24.0603409Z", "durationInMs": 2.7956},{"event": "Received", "startTimeUtc": "2026-04-14T17:53:24.0631365Z", "durationInMs": 0.1599},{"event": "Completed", "startTimeUtc": "2026-04-14T17:53:24.0632964Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:53:24.0507169Z","lastSend":"2026-04-14T17:53:24.0507169Z","lastReceive":"2026-04-14T17:53:24.0507169Z"},"requestSizeInBytes":1638,"requestBodySizeInBytes":1188,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:53:24.0194664Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"ec325b69-1c99-4846-9e94-594881c5251e","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:53:24.0194664Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0f173161-b0b4-4891-8a20-355829cb6acd","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:53:24.0194664Z","DurationInMs":15.6806,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"aaae6ead-1604-48b4-94b8-8b8c63970d5b","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.34376629999999997,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"aaae6ead-1604-48b4-94b8-8b8c63970d5b","ResponseTimeUtc":"2026-04-14T17:53:24.0507169Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: aaae6ead-1604-48b4-94b8-8b8c63970d5b, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149711","BELatencyInMs":"0.598"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 17:59:09 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 02d63ed2-b95e-43b4-a83d-caeebf843ae6; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 02d63ed2-b95e-43b4-a83d-caeebf843ae6, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 02d63ed2-b95e-43b4-a83d-caeebf843ae6; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 02d63ed2-b95e-43b4-a83d-caeebf843ae6, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T17:59:09.772Z","duration in milliseconds":45.676700000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T17:59:09.7101384Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92032,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0514},{"name":"Get PkValue From Stream","duration in milliseconds":6.4881,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003},{"name":"Read Collection","duration in milliseconds":6.3224,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:59:09.7726367Z","DurationInMs":15.6263,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"4a036e56-10b4-4bc1-9051-92e0f3909973","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":38.9517,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":38.9162,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:58:17.6215135Z","cpu":2.500,"memory":5275540.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1056,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:58:27.6331434Z","cpu":1.482,"memory":5302076.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0892,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:58:37.6365478Z","cpu":0.781,"memory":5303912.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0891,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:58:47.6443488Z","cpu":0.624,"memory":5300844.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0886,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:58:57.6459148Z","cpu":0.703,"memory":5294980.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1064,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:59:07.6608342Z","cpu":3.198,"memory":5244216.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1215,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":38.905,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":38.9016,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":38.880700000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":38.8782,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":37.8889,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T17:59:09.7882630Z","EndTimeUTC":"2026-04-14T17:59:09.8038871Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T17:59:09.8195113Z","DurationInMs":15.6242,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"02d63ed2-b95e-43b4-a83d-caeebf843ae6","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149711,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149711,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149711,"SessionToken":"-1#26149711","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.686","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 5:59:09 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T17:59:09.8038871Z", "durationInMs": 0.0096},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T17:59:09.8038967Z", "durationInMs": 20.996},{"event": "Pipelined", "startTimeUtc": "2026-04-14T17:59:09.8248927Z", "durationInMs": 0.2125},{"event": "Transit Time", "startTimeUtc": "2026-04-14T17:59:09.8251052Z", "durationInMs": 2.1181},{"event": "Received", "startTimeUtc": "2026-04-14T17:59:09.8272233Z", "durationInMs": 0.1505},{"event": "Completed", "startTimeUtc": "2026-04-14T17:59:09.8273738Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T17:59:09.8195113Z","lastSend":"2026-04-14T17:59:09.8195113Z","lastReceive":"2026-04-14T17:59:09.8195113Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1186,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T17:59:09.7882630Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"2cf79dc8-fa7a-4352-a8da-abdf70346fd7","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T17:59:09.7882630Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"7eff42d6-4ed3-40cb-8572-2b4135f0a8f2","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T17:59:09.7882630Z","DurationInMs":15.6241,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"02d63ed2-b95e-43b4-a83d-caeebf843ae6","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1093729,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"02d63ed2-b95e-43b4-a83d-caeebf843ae6","ResponseTimeUtc":"2026-04-14T17:59:09.8195113Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 02d63ed2-b95e-43b4-a83d-caeebf843ae6, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149711","BELatencyInMs":"0.686"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001}]}
2026-04-14 18:00:14 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 27fbd0d1-a68f-4cb9-8f83-37c5c972e884; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 27fbd0d1-a68f-4cb9-8f83-37c5c972e884, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 27fbd0d1-a68f-4cb9-8f83-37c5c972e884; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 27fbd0d1-a68f-4cb9-8f83-37c5c972e884, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:00:14.487Z","duration in milliseconds":41.8054,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:00:13.6902504Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92035,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0397},{"name":"Get PkValue From Stream","duration in milliseconds":6.5702000000000007,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0034000000000000002},{"name":"Read Collection","duration in milliseconds":6.4323000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:00:14.4871321Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"1e570e98-2bf0-4079-b789-387d66ec1dce","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.0255,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.0003,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:59:17.6745543Z","cpu":3.432,"memory":5243264.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.094,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:59:27.6773745Z","cpu":1.641,"memory":5239544.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2367,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:59:37.6832780Z","cpu":3.672,"memory":5238608.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1136,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:59:47.6848039Z","cpu":0.547,"memory":5240864.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2651,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:59:57.6996909Z","cpu":1.170,"memory":5240344.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0861,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:00:07.7097811Z","cpu":9.438,"memory":5228828.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0876,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":34.9943,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":34.9894,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":34.9688,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":34.9652,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":33.919000000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:00:14.5032051Z","EndTimeUTC":"2026-04-14T18:00:14.5032051Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:00:14.5183850Z","DurationInMs":15.1799,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"27fbd0d1-a68f-4cb9-8f83-37c5c972e884","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149711,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149711,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149711,"SessionToken":"-1#26149711","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.615","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:00:14 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:00:14.5032051Z", "durationInMs": 0.0079},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:00:14.5032130Z", "durationInMs": 16.4686},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:00:14.5196816Z", "durationInMs": 0.1811},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:00:14.5198627Z", "durationInMs": 1.8257},{"event": "Received", "startTimeUtc": "2026-04-14T18:00:14.5216884Z", "durationInMs": 0.1227},{"event": "Completed", "startTimeUtc": "2026-04-14T18:00:14.5218111Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:00:14.5183850Z","lastSend":"2026-04-14T18:00:14.5183850Z","lastReceive":"2026-04-14T18:00:14.5183850Z"},"requestSizeInBytes":1602,"requestBodySizeInBytes":1146,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:00:14.4871321Z","DurationInMs":16.073,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"aea398e9-9fb6-401b-bd75-1c1c6793572d","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:00:14.5032051Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"04f6fa57-ca7d-4a58-be9e-e3d003d7af86","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:00:14.5032051Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"27fbd0d1-a68f-4cb9-8f83-37c5c972e884","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.82813459999999994,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"27fbd0d1-a68f-4cb9-8f83-37c5c972e884","ResponseTimeUtc":"2026-04-14T18:00:14.5340117Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 27fbd0d1-a68f-4cb9-8f83-37c5c972e884, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149711","BELatencyInMs":"0.615"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0035}]}
2026-04-14 18:00:38 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 43f704eb-6ad5-47e9-9907-ba00aced0ba8; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 43f704eb-6ad5-47e9-9907-ba00aced0ba8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 43f704eb-6ad5-47e9-9907-ba00aced0ba8; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 43f704eb-6ad5-47e9-9907-ba00aced0ba8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:00:38.765Z","duration in milliseconds":42.8451,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:00:38.6868442Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92040,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0555},{"name":"Get PkValue From Stream","duration in milliseconds":6.3437,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003},{"name":"Read Collection","duration in milliseconds":6.1723,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:00:38.7650400Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"844deb06-87ff-4601-8c5a-fcda3dafaa5d","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.225100000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.181200000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T17:59:47.6848039Z","cpu":0.547,"memory":5240864.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2651,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T17:59:57.6996909Z","cpu":1.170,"memory":5240344.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0861,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:00:07.7097811Z","cpu":9.438,"memory":5228828.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0876,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:00:17.7245081Z","cpu":4.524,"memory":5168268.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1265,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:00:27.7348838Z","cpu":1.794,"memory":5166544.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1301,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:00:37.7457945Z","cpu":2.734,"memory":5166344.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1093,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.172000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.1667,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.144200000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.141,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.099900000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:00:38.7805946Z","EndTimeUTC":"2026-04-14T18:00:38.7805946Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:00:38.7962301Z","DurationInMs":15.6355,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"43f704eb-6ad5-47e9-9907-ba00aced0ba8","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149711,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149711,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149711,"SessionToken":"-1#26149711","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.539","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:00:38 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:00:38.7805946Z", "durationInMs": 0.0079},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:00:38.7806025Z", "durationInMs": 17.9203},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:00:38.7985228Z", "durationInMs": 0.2312},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:00:38.7987540Z", "durationInMs": 1.6936},{"event": "Received", "startTimeUtc": "2026-04-14T18:00:38.8004476Z", "durationInMs": 0.1355},{"event": "Completed", "startTimeUtc": "2026-04-14T18:00:38.8005831Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:00:38.7962301Z","lastSend":"2026-04-14T18:00:38.7962301Z","lastReceive":"2026-04-14T18:00:38.7962301Z"},"requestSizeInBytes":1633,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:00:38.7650400Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0650bf9e-a767-491b-b5d6-00582197774c","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:00:38.7650400Z","DurationInMs":15.5546,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"ff310aa6-dc74-4c64-aeb3-5ce28311ee7d","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:00:38.7805946Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"43f704eb-6ad5-47e9-9907-ba00aced0ba8","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1093859,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"43f704eb-6ad5-47e9-9907-ba00aced0ba8","ResponseTimeUtc":"2026-04-14T18:00:38.7962301Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 43f704eb-6ad5-47e9-9907-ba00aced0ba8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149711","BELatencyInMs":"0.539"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0027}]}
2026-04-14 18:01:03 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: fadff687-db28-4dba-b385-0c5f1bfd4aae; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: fadff687-db28-4dba-b385-0c5f1bfd4aae, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: fadff687-db28-4dba-b385-0c5f1bfd4aae; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: fadff687-db28-4dba-b385-0c5f1bfd4aae, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:01:03.045Z","duration in milliseconds":42.7107,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:01:02.9671144Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92042,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0528},{"name":"Get PkValue From Stream","duration in milliseconds":7.2294,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001},{"name":"Read Collection","duration in milliseconds":7.0451000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:01:03.0452389Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"10b30d9b-d18c-4c87-a2c1-958a3d966ec3","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.237500000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.191700000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:00:07.7097811Z","cpu":9.438,"memory":5228828.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0876,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:00:17.7245081Z","cpu":4.524,"memory":5168268.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1265,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:00:27.7348838Z","cpu":1.794,"memory":5166544.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1301,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:00:37.7457945Z","cpu":2.734,"memory":5166344.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1093,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:00:47.7517277Z","cpu":1.716,"memory":5165688.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0669,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:00:57.7623991Z","cpu":2.109,"memory":5166864.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0742,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.1843,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.1802,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.1571,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.1539,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.284400000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:01:03.0608760Z","EndTimeUTC":"2026-04-14T18:01:03.0608760Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:01:03.0921169Z","DurationInMs":31.2409,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"fadff687-db28-4dba-b385-0c5f1bfd4aae","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149711,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149711,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149711,"SessionToken":"-1#26149711","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.672","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:01:03 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:01:03.0608760Z", "durationInMs": 0.0088},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:01:03.0608848Z", "durationInMs": 16.9045},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:01:03.0777893Z", "durationInMs": 0.1963},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:01:03.0779856Z", "durationInMs": 1.8605},{"event": "Received", "startTimeUtc": "2026-04-14T18:01:03.0798461Z", "durationInMs": 0.1371},{"event": "Completed", "startTimeUtc": "2026-04-14T18:01:03.0799832Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:01:03.0764915Z","lastSend":"2026-04-14T18:01:03.0764915Z","lastReceive":"2026-04-14T18:01:03.0764915Z"},"requestSizeInBytes":1635,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:01:03.0608760Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"358d1b5b-33d3-407f-aa4a-aefe2f90f0d4","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:01:03.0608760Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"1c6edf6e-853e-427a-ba42-e162580733f5","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:01:03.0608760Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"fadff687-db28-4dba-b385-0c5f1bfd4aae","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1250025,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"fadff687-db28-4dba-b385-0c5f1bfd4aae","ResponseTimeUtc":"2026-04-14T18:01:03.0921169Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: fadff687-db28-4dba-b385-0c5f1bfd4aae, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149711","BELatencyInMs":"0.672"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 18:01:34 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 81bd008d-f0be-461b-bae3-a7818e3f35f5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 81bd008d-f0be-461b-bae3-a7818e3f35f5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 81bd008d-f0be-461b-bae3-a7818e3f35f5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 81bd008d-f0be-461b-bae3-a7818e3f35f5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:01:34.223Z","duration in milliseconds":41.771100000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:01:34.1447997Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92048,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0543},{"name":"Get PkValue From Stream","duration in milliseconds":6.4628000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0045000000000000005},{"name":"Read Collection","duration in milliseconds":6.2538,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:01:34.2230051Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"bcd5c2b5-8c01-421f-8c75-c359f658f48e","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.0692,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.0308,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:00:37.7457945Z","cpu":2.734,"memory":5166344.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1093,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:00:47.7517277Z","cpu":1.716,"memory":5165688.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0669,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:00:57.7623991Z","cpu":2.109,"memory":5166864.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0742,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:01:07.7653677Z","cpu":4.219,"memory":5177024.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1058,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:01:17.7816567Z","cpu":1.950,"memory":5177216.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2941,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:01:27.7853489Z","cpu":2.028,"memory":5176960.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1011,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.0221,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.017700000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":34.998200000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":34.9955,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.034600000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:01:34.2386340Z","EndTimeUTC":"2026-04-14T18:01:34.2386340Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:01:34.2541771Z","DurationInMs":15.5431,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"81bd008d-f0be-461b-bae3-a7818e3f35f5","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149711,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149711,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149711,"SessionToken":"-1#26149711","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.506","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:01:34 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:01:34.2386340Z", "durationInMs": 0.0097},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:01:34.2386437Z", "durationInMs": 17.4377},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:01:34.2560814Z", "durationInMs": 0.1917},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:01:34.2562731Z", "durationInMs": 1.7472},{"event": "Received", "startTimeUtc": "2026-04-14T18:01:34.2580203Z", "durationInMs": 0.1271},{"event": "Completed", "startTimeUtc": "2026-04-14T18:01:34.2581474Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:01:34.2541771Z","lastSend":"2026-04-14T18:01:34.2541771Z","lastReceive":"2026-04-14T18:01:34.2541771Z"},"requestSizeInBytes":1637,"requestBodySizeInBytes":1187,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:01:34.2230051Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"7a5b1c22-eda7-4faa-b4bb-3aeb2c580568","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:01:34.2230051Z","DurationInMs":15.628900000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a0c3a2ed-e20e-4e88-a116-d20ffb62fc98","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:01:34.2386340Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"81bd008d-f0be-461b-bae3-a7818e3f35f5","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1093774,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"81bd008d-f0be-461b-bae3-a7818e3f35f5","ResponseTimeUtc":"2026-04-14T18:01:34.2541771Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 81bd008d-f0be-461b-bae3-a7818e3f35f5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149711","BELatencyInMs":"0.506"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 18:04:06 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 73c520d2-436f-4354-b2a5-bc2b17db5da9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 73c520d2-436f-4354-b2a5-bc2b17db5da9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 73c520d2-436f-4354-b2a5-bc2b17db5da9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 73c520d2-436f-4354-b2a5-bc2b17db5da9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:04:06.231Z","duration in milliseconds":61.936400000000006,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:04:06.1376542Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92080,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0524},{"name":"Get PkValue From Stream","duration in milliseconds":21.7632,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002},{"name":"Read Collection","duration in milliseconds":21.584,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:04:06.2313984Z","DurationInMs":15.7042,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"61153de7-0cfe-4615-a1aa-b71e4c4d3172","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":39.9037,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":39.860800000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:03:07.8606548Z","cpu":2.031,"memory":5290260.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1406,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:03:17.8674496Z","cpu":1.404,"memory":5283744.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1639,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:03:27.8793370Z","cpu":2.262,"memory":5244212.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0564,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:03:37.8837003Z","cpu":1.719,"memory":5242716.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1369,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:03:47.8932434Z","cpu":0.391,"memory":5247448.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0656,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:03:57.8977766Z","cpu":1.950,"memory":5248620.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0845,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":39.8513,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":39.8468,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":39.8205,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":39.8183,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":37.956,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:04:06.2627045Z","EndTimeUTC":"2026-04-14T18:04:06.2627045Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:04:06.2939024Z","DurationInMs":31.1979,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"73c520d2-436f-4354-b2a5-bc2b17db5da9","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149711,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149711,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149711,"SessionToken":"-1#26149711","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.603","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:04:06 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:04:06.2627045Z", "durationInMs": 0.0094},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:04:06.2627139Z", "durationInMs": 17.7943},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:04:06.2805082Z", "durationInMs": 0.208},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:04:06.2807162Z", "durationInMs": 3.2585},{"event": "Received", "startTimeUtc": "2026-04-14T18:04:06.2839747Z", "durationInMs": 0.178},{"event": "Completed", "startTimeUtc": "2026-04-14T18:04:06.2841527Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:04:06.2782763Z","lastSend":"2026-04-14T18:04:06.2782763Z","lastReceive":"2026-04-14T18:04:06.2782763Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:04:06.2471026Z","DurationInMs":15.6019,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"4b384723-164e-46ec-896d-8963b6373c47","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:04:06.2627045Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"c23a6cf3-3fff-472f-b6b7-42bd12a815f9","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:04:06.2627045Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"73c520d2-436f-4354-b2a5-bc2b17db5da9","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1562482,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"73c520d2-436f-4354-b2a5-bc2b17db5da9","ResponseTimeUtc":"2026-04-14T18:04:06.2939024Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 73c520d2-436f-4354-b2a5-bc2b17db5da9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149711","BELatencyInMs":"0.603"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 18:06:27 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 39118ae9-ca21-40ff-99aa-db6420371a81; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 39118ae9-ca21-40ff-99aa-db6420371a81, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 39118ae9-ca21-40ff-99aa-db6420371a81; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 39118ae9-ca21-40ff-99aa-db6420371a81, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:06:27.068Z","duration in milliseconds":43.0197,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:06:26.5838225Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92099,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.048400000000000006},{"name":"Get PkValue From Stream","duration in milliseconds":6.2729,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0039000000000000003},{"name":"Read Collection","duration in milliseconds":6.1084000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:06:27.0682010Z","DurationInMs":15.6858,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"8ab8a4b6-b2fd-414d-a775-11dd8e1ce0da","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.520900000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.487700000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:05:27.9811628Z","cpu":1.641,"memory":5201204.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1129,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:05:37.9889700Z","cpu":2.891,"memory":5196356.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0971,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:05:47.9945418Z","cpu":4.368,"memory":5196032.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0914,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:05:58.0053839Z","cpu":2.109,"memory":5155960.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2663,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:06:08.0211919Z","cpu":2.340,"memory":5161440.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1126,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:06:18.0371151Z","cpu":1.404,"memory":5167492.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1343,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.4793,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.475,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.4534,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.449600000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.493500000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:06:27.0838868Z","EndTimeUTC":"2026-04-14T18:06:27.0995115Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:06:27.1150826Z","DurationInMs":15.571100000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"39118ae9-ca21-40ff-99aa-db6420371a81","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149711,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149711,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149711,"SessionToken":"-1#26149711","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.888","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:06:27 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:06:27.0995115Z", "durationInMs": 0.0087},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:06:27.0995202Z", "durationInMs": 18.3043},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:06:27.1178245Z", "durationInMs": 0.2035},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:06:27.1180280Z", "durationInMs": 2.1137},{"event": "Received", "startTimeUtc": "2026-04-14T18:06:27.1201417Z", "durationInMs": 0.1401},{"event": "Completed", "startTimeUtc": "2026-04-14T18:06:27.1202818Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:06:27.1150826Z","lastSend":"2026-04-14T18:06:27.1150826Z","lastReceive":"2026-04-14T18:06:27.1150826Z"},"requestSizeInBytes":1599,"requestBodySizeInBytes":1149,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:06:27.0838868Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"5996e9c7-b6a0-4ee1-8504-fceff7cd68da","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:06:27.0838868Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"121e64f8-eae5-4cca-acb7-0715cae46fdc","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:06:27.0838868Z","DurationInMs":15.6247,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"39118ae9-ca21-40ff-99aa-db6420371a81","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.5312601,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"39118ae9-ca21-40ff-99aa-db6420371a81","ResponseTimeUtc":"2026-04-14T18:06:27.1150826Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 39118ae9-ca21-40ff-99aa-db6420371a81, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149711","BELatencyInMs":"0.888"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 18:08:36 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 00cb3f6e-50a9-40eb-ac38-7a293664c29d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 00cb3f6e-50a9-40eb-ac38-7a293664c29d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 00cb3f6e-50a9-40eb-ac38-7a293664c29d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 00cb3f6e-50a9-40eb-ac38-7a293664c29d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:08:36.789Z","duration in milliseconds":44.9579,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:08:36.7265725Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92111,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.051000000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":6.4842,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0016},{"name":"Read Collection","duration in milliseconds":6.3074,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:08:36.7890798Z","DurationInMs":15.617600000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"22bc0692-fa19-4e92-ba82-e5c6001bdcb2","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":38.2492,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":38.2151,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:07:38.1268303Z","cpu":0.780,"memory":5236552.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1023,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:07:48.1357134Z","cpu":1.014,"memory":5241640.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.087,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:07:58.1377491Z","cpu":3.516,"memory":5248016.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0784,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:08:08.1412096Z","cpu":1.406,"memory":5319184.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.093,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:08:18.1568371Z","cpu":2.808,"memory":5313048.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0981,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:08:28.1715445Z","cpu":0.624,"memory":5326088.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.075,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":38.2077,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":38.2044,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":38.1864,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":38.1839,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":37.2098,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:08:36.8203239Z","EndTimeUTC":"2026-04-14T18:08:36.8203239Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:08:36.8359470Z","DurationInMs":15.6231,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"00cb3f6e-50a9-40eb-ac38-7a293664c29d","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149711,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149711,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149711,"SessionToken":"-1#26149711","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.615","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:08:36 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:08:36.8203239Z", "durationInMs": 0.0098},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:08:36.8203337Z", "durationInMs": 20.2272},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:08:36.8405609Z", "durationInMs": 0.2055},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:08:36.8407664Z", "durationInMs": 1.9847},{"event": "Received", "startTimeUtc": "2026-04-14T18:08:36.8427511Z", "durationInMs": 0.1341},{"event": "Completed", "startTimeUtc": "2026-04-14T18:08:36.8428852Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:08:36.8359470Z","lastSend":"2026-04-14T18:08:36.8359470Z","lastReceive":"2026-04-14T18:08:36.8359470Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:08:36.8046974Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"8a5d3d8c-c868-40ec-aca4-78aae82c1896","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:08:36.8046974Z","DurationInMs":15.6265,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"f9349d68-9fd1-4be1-aa3f-98e9d1446275","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:08:36.8203239Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"00cb3f6e-50a9-40eb-ac38-7a293664c29d","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1093745,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"00cb3f6e-50a9-40eb-ac38-7a293664c29d","ResponseTimeUtc":"2026-04-14T18:08:36.8359470Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 00cb3f6e-50a9-40eb-ac38-7a293664c29d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149711","BELatencyInMs":"0.615"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 18:11:48 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e6040683-50e4-44d4-9d7e-79b58f92d7df; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e6040683-50e4-44d4-9d7e-79b58f92d7df, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e6040683-50e4-44d4-9d7e-79b58f92d7df; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e6040683-50e4-44d4-9d7e-79b58f92d7df, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:11:48.508Z","duration in milliseconds":45.4095,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:11:48.3992358Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92138,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0478},{"name":"Get PkValue From Stream","duration in milliseconds":6.0696,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0015},{"name":"Read Collection","duration in milliseconds":5.905,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:11:48.5086133Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"aaae1ff3-a365-4327-a002-5825eeeb3500","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":39.1021,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":39.0668,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:10:58.3342483Z","cpu":4.446,"memory":5227516.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0432,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:11:08.3499345Z","cpu":3.978,"memory":5238168.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0765,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:11:18.3514080Z","cpu":5.547,"memory":5237116.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2355,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:11:28.3605470Z","cpu":1.014,"memory":5237400.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1169,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:11:38.3761976Z","cpu":4.680,"memory":5240048.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0778,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:11:48.3836770Z","cpu":4.531,"memory":5240316.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1132,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":39.0585,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":39.0535,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":39.031400000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":39.0281,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":38.0596,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:11:48.5242411Z","EndTimeUTC":"2026-04-14T18:11:48.5242411Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:11:48.5554888Z","DurationInMs":31.247700000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"e6040683-50e4-44d4-9d7e-79b58f92d7df","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149714,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149714,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149714,"SessionToken":"-1#26149714","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"1.021","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:11:48 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:11:48.5242411Z", "durationInMs": 0.0086},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:11:48.5242497Z", "durationInMs": 19.9707},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:11:48.5442204Z", "durationInMs": 0.2113},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:11:48.5444317Z", "durationInMs": 2.6046},{"event": "Received", "startTimeUtc": "2026-04-14T18:11:48.5470363Z", "durationInMs": 0.1428},{"event": "Completed", "startTimeUtc": "2026-04-14T18:11:48.5471791Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:11:48.5398668Z","lastSend":"2026-04-14T18:11:48.5398668Z","lastReceive":"2026-04-14T18:11:48.5398668Z"},"requestSizeInBytes":1628,"requestBodySizeInBytes":1178,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:11:48.5086133Z","DurationInMs":15.6278,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a43d93be-6d84-480d-8914-26adec3a3993","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:11:48.5242411Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"74dbffca-bb4b-4447-9377-0690f340561f","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:11:48.5242411Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"e6040683-50e4-44d4-9d7e-79b58f92d7df","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.156253,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"e6040683-50e4-44d4-9d7e-79b58f92d7df","ResponseTimeUtc":"2026-04-14T18:11:48.5554888Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: e6040683-50e4-44d4-9d7e-79b58f92d7df, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149714","BELatencyInMs":"1.021"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 18:14:16 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 7414d6b0-e4ee-4952-a8fe-7581d8608eac; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 7414d6b0-e4ee-4952-a8fe-7581d8608eac, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 7414d6b0-e4ee-4952-a8fe-7581d8608eac; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 7414d6b0-e4ee-4952-a8fe-7581d8608eac, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:14:16.198Z","duration in milliseconds":51.2729,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:14:16.0891345Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92160,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0432},{"name":"Get PkValue From Stream","duration in milliseconds":6.0995,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.001},{"name":"Read Collection","duration in milliseconds":5.9524,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:14:16.1985664Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"63559bcf-fa4a-4997-bf24-eb770a92ce6b","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":44.956700000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":44.926700000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:13:18.4651826Z","cpu":1.014,"memory":5289872.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0815,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:13:28.4675203Z","cpu":0.469,"memory":5288480.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0668,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:13:38.4771891Z","cpu":1.560,"memory":5282380.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1204,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:13:48.4921797Z","cpu":3.354,"memory":5286876.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0986,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:13:58.4991062Z","cpu":1.484,"memory":5292916.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0727,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:14:08.5003574Z","cpu":0.936,"memory":5289276.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1192,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":44.9193,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":44.9151,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":44.8941,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":44.8913,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":43.9392,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:14:16.2141348Z","EndTimeUTC":"2026-04-14T18:14:16.2141348Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:14:16.2453814Z","DurationInMs":31.2466,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"7414d6b0-e4ee-4952-a8fe-7581d8608eac","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149714,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149714,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149714,"SessionToken":"-1#26149714","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.567","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:14:16 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:14:16.2141348Z", "durationInMs": 0.0094},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:14:16.2141442Z", "durationInMs": 27.8398},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:14:16.2419840Z", "durationInMs": 0.1875},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:14:16.2421715Z", "durationInMs": 1.9482},{"event": "Received", "startTimeUtc": "2026-04-14T18:14:16.2441197Z", "durationInMs": 0.1397},{"event": "Completed", "startTimeUtc": "2026-04-14T18:14:16.2442594Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:14:16.2453814Z","lastSend":"2026-04-14T18:14:16.2453814Z","lastReceive":"2026-04-14T18:14:16.2453814Z"},"requestSizeInBytes":1633,"requestBodySizeInBytes":1183,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:14:16.1985664Z","DurationInMs":15.5684,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"1e3e075c-3163-46a7-a3e0-b80a79af4780","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:14:16.2141348Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"e1e5ed5b-d1d9-4996-8afb-15f4866490a2","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:14:16.2141348Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"7414d6b0-e4ee-4952-a8fe-7581d8608eac","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1562469,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"7414d6b0-e4ee-4952-a8fe-7581d8608eac","ResponseTimeUtc":"2026-04-14T18:14:16.2453814Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 7414d6b0-e4ee-4952-a8fe-7581d8608eac, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149714","BELatencyInMs":"0.567"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0015}]}
2026-04-14 18:14:28 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 4ee04d9e-c0df-4b87-aefe-3931fab6802a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 4ee04d9e-c0df-4b87-aefe-3931fab6802a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 4ee04d9e-c0df-4b87-aefe-3931fab6802a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 4ee04d9e-c0df-4b87-aefe-3931fab6802a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:14:28.921Z","duration in milliseconds":45.9067,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:14:28.7812877Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92163,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0374},{"name":"Get PkValue From Stream","duration in milliseconds":6.4042,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":6.2616000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:14:28.9219082Z","DurationInMs":15.6293,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"5e9b762f-6508-4d80-99b1-6f69676fe14d","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":39.303000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":39.272200000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:13:38.4771891Z","cpu":1.560,"memory":5282380.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1204,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:13:48.4921797Z","cpu":3.354,"memory":5286876.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0986,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:13:58.4991062Z","cpu":1.484,"memory":5292916.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0727,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:14:08.5003574Z","cpu":0.936,"memory":5289276.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1192,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:14:18.5078878Z","cpu":2.266,"memory":5287412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2509,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:14:28.5157292Z","cpu":1.563,"memory":5289088.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.065,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":39.264700000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":39.2616,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":39.2446,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":39.2426,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":38.414100000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:14:28.9375375Z","EndTimeUTC":"2026-04-14T18:14:28.9375375Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:14:28.9687838Z","DurationInMs":31.2463,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"4ee04d9e-c0df-4b87-aefe-3931fab6802a","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149714,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149714,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149714,"SessionToken":"-1#26149714","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.686","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:14:28 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:14:28.9375375Z", "durationInMs": 0.0079},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:14:28.9375454Z", "durationInMs": 19.2559},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:14:28.9568013Z", "durationInMs": 0.1679},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:14:28.9569692Z", "durationInMs": 4.6961},{"event": "Received", "startTimeUtc": "2026-04-14T18:14:28.9616653Z", "durationInMs": 0.1277},{"event": "Completed", "startTimeUtc": "2026-04-14T18:14:28.9617930Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:14:28.9531587Z","lastSend":"2026-04-14T18:14:28.9531587Z","lastReceive":"2026-04-14T18:14:28.9687838Z"},"requestSizeInBytes":1633,"requestBodySizeInBytes":1183,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:14:28.9375375Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"d254ed81-0caa-4166-898d-44a0ed319509","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:14:28.9375375Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0c2ba2f5-4ab8-4470-a63f-6f34321fd69b","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:14:28.9375375Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"4ee04d9e-c0df-4b87-aefe-3931fab6802a","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1874961,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"4ee04d9e-c0df-4b87-aefe-3931fab6802a","ResponseTimeUtc":"2026-04-14T18:14:28.9687838Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 4ee04d9e-c0df-4b87-aefe-3931fab6802a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149714","BELatencyInMs":"0.686"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0025}]}
2026-04-14 18:14:55 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: af59ffef-9cc6-4761-b874-bf9ec9b4b887; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: af59ffef-9cc6-4761-b874-bf9ec9b4b887, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: af59ffef-9cc6-4761-b874-bf9ec9b4b887; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: af59ffef-9cc6-4761-b874-bf9ec9b4b887, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:14:55.324Z","duration in milliseconds":42.9981,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:14:55.1838541Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92172,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0396},{"name":"Get PkValue From Stream","duration in milliseconds":5.9968,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014},{"name":"Read Collection","duration in milliseconds":5.8297,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:14:55.3244756Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"ab42b7a6-d3ec-4872-9125-ad34685508aa","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.7765,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.7419,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:13:58.4991062Z","cpu":1.484,"memory":5292916.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0727,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:14:08.5003574Z","cpu":0.936,"memory":5289276.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1192,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:14:18.5078878Z","cpu":2.266,"memory":5287412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2509,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:14:28.5157292Z","cpu":1.563,"memory":5289088.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.065,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:14:38.5245529Z","cpu":3.276,"memory":5281084.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1056,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:14:48.5379249Z","cpu":9.516,"memory":5264840.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0996,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.734700000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.7302,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.711,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.7078,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.7385,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:14:55.3401012Z","EndTimeUTC":"2026-04-14T18:14:55.3401012Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:14:55.3557295Z","DurationInMs":15.628300000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"af59ffef-9cc6-4761-b874-bf9ec9b4b887","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149714,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149714,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149714,"SessionToken":"-1#26149714","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.561","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:14:55 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:14:55.3401012Z", "durationInMs": 0.0088},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:14:55.3401100Z", "durationInMs": 19.3854},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:14:55.3594954Z", "durationInMs": 0.2209},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:14:55.3597163Z", "durationInMs": 2.0797},{"event": "Received", "startTimeUtc": "2026-04-14T18:14:55.3617960Z", "durationInMs": 0.1413},{"event": "Completed", "startTimeUtc": "2026-04-14T18:14:55.3619373Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:14:55.3557295Z","lastSend":"2026-04-14T18:14:55.3557295Z","lastReceive":"2026-04-14T18:14:55.3557295Z"},"requestSizeInBytes":1631,"requestBodySizeInBytes":1183,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:14:55.3244756Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"3289d1b9-fe00-4f94-b81c-0bcfa0f1bf32","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:14:55.3244756Z","DurationInMs":15.6256,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"1acdb17c-cf2d-42a5-817e-3440ceafb27a","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:14:55.3401012Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"af59ffef-9cc6-4761-b874-bf9ec9b4b887","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.17187539999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"af59ffef-9cc6-4761-b874-bf9ec9b4b887","ResponseTimeUtc":"2026-04-14T18:14:55.3557295Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: af59ffef-9cc6-4761-b874-bf9ec9b4b887, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149714","BELatencyInMs":"0.561"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.003}]}
2026-04-14 18:15:47 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 279dc273-d99c-4817-90f1-fe892b731a3d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 279dc273-d99c-4817-90f1-fe892b731a3d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 279dc273-d99c-4817-90f1-fe892b731a3d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 279dc273-d99c-4817-90f1-fe892b731a3d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:15:47.480Z","duration in milliseconds":43.9088,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:15:47.3558489Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92177,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0548},{"name":"Get PkValue From Stream","duration in milliseconds":6.6924,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0016},{"name":"Read Collection","duration in milliseconds":6.5348000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:15:47.4808526Z","DurationInMs":15.696900000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"b595f570-b4bc-4e11-80c0-01f2588d7342","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.9752,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.9346,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:14:48.5379249Z","cpu":9.516,"memory":5264840.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0996,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:14:58.5471273Z","cpu":3.281,"memory":5258568.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0874,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:15:08.5520633Z","cpu":9.360,"memory":5245896.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1158,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:15:18.5596127Z","cpu":2.109,"memory":5235696.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0969,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:15:28.5689994Z","cpu":2.340,"memory":5235220.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.061,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:15:38.5748309Z","cpu":0.859,"memory":5234224.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.062,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.9257,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.919200000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.8966,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.891200000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.018100000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:15:47.4965495Z","EndTimeUTC":"2026-04-14T18:15:47.5121770Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:15:47.5277282Z","DurationInMs":15.551200000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"279dc273-d99c-4817-90f1-fe892b731a3d","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149714,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149714,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149714,"SessionToken":"-1#26149714","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.91","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:15:47 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:15:47.5121770Z", "durationInMs": 0.0085},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:15:47.5121855Z", "durationInMs": 18.3259},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:15:47.5305114Z", "durationInMs": 0.2043},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:15:47.5307157Z", "durationInMs": 2.4516},{"event": "Received", "startTimeUtc": "2026-04-14T18:15:47.5331673Z", "durationInMs": 0.1523},{"event": "Completed", "startTimeUtc": "2026-04-14T18:15:47.5333196Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:15:47.5277282Z","lastSend":"2026-04-14T18:15:47.5277282Z","lastReceive":"2026-04-14T18:15:47.5277282Z"},"requestSizeInBytes":1633,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:15:47.4965495Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"43fe9fa4-8763-4122-9ef6-338a2b507f03","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:15:47.4965495Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0ed69b91-33e9-4d71-be9b-3ffdca93622b","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:15:47.4965495Z","DurationInMs":15.627500000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"279dc273-d99c-4817-90f1-fe892b731a3d","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.17187929999999999,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"279dc273-d99c-4817-90f1-fe892b731a3d","ResponseTimeUtc":"2026-04-14T18:15:47.5277282Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 279dc273-d99c-4817-90f1-fe892b731a3d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149714","BELatencyInMs":"0.91"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 18:15:59 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b20bdaad-84e2-4c30-960a-4971a96c40d8; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b20bdaad-84e2-4c30-960a-4971a96c40d8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b20bdaad-84e2-4c30-960a-4971a96c40d8; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b20bdaad-84e2-4c30-960a-4971a96c40d8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:15:59.061Z","duration in milliseconds":41.784,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:15:58.9205491Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92181,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0555},{"name":"Get PkValue From Stream","duration in milliseconds":7.5708,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0056},{"name":"Read Collection","duration in milliseconds":7.3862000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:15:59.0611876Z","DurationInMs":15.6994,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"90cc041b-4a7a-4a4a-b6b7-5d48f28aa3bd","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":33.9724,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":33.9367,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:15:08.5520633Z","cpu":9.360,"memory":5245896.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1158,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:15:18.5596127Z","cpu":2.109,"memory":5235696.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0969,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:15:28.5689994Z","cpu":2.340,"memory":5235220.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.061,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:15:38.5748309Z","cpu":0.859,"memory":5234224.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.062,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:15:48.5902520Z","cpu":2.808,"memory":5233372.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.121,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:15:58.5925020Z","cpu":4.531,"memory":5192172.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.068,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":33.8387,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":33.8325,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":33.8048,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":33.8006,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":32.9018,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:15:59.0768870Z","EndTimeUTC":"2026-04-14T18:15:59.0924302Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:15:59.1080560Z","DurationInMs":15.6258,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"b20bdaad-84e2-4c30-960a-4971a96c40d8","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149714,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149714,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149714,"SessionToken":"-1#26149714","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.72","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:15:59 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:15:59.0924302Z", "durationInMs": 0.0072},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:15:59.0924374Z", "durationInMs": 15.2043},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:15:59.1076417Z", "durationInMs": 0.2078},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:15:59.1078495Z", "durationInMs": 2.1152},{"event": "Received", "startTimeUtc": "2026-04-14T18:15:59.1099647Z", "durationInMs": 0.1396},{"event": "Completed", "startTimeUtc": "2026-04-14T18:15:59.1101043Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:15:59.1080560Z","lastSend":"2026-04-14T18:15:59.1080560Z","lastReceive":"2026-04-14T18:15:59.1080560Z"},"requestSizeInBytes":1635,"requestBodySizeInBytes":1183,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:15:59.0768870Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"4c62c897-ce1f-4c79-b98a-5b5948637a6a","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:15:59.0768870Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"140fd7c1-0610-44f8-a3bc-524e419d910d","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:15:59.0768870Z","DurationInMs":15.5432,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"b20bdaad-84e2-4c30-960a-4971a96c40d8","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1875069,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"b20bdaad-84e2-4c30-960a-4971a96c40d8","ResponseTimeUtc":"2026-04-14T18:15:59.1080560Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: b20bdaad-84e2-4c30-960a-4971a96c40d8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149714","BELatencyInMs":"0.72"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 18:16:13 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e374f5a5-c260-432e-b409-5ee4811db155; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e374f5a5-c260-432e-b409-5ee4811db155, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e374f5a5-c260-432e-b409-5ee4811db155; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e374f5a5-c260-432e-b409-5ee4811db155, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:16:13.711Z","duration in milliseconds":42.0966,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:16:13.5551684Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92184,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0386},{"name":"Get PkValue From Stream","duration in milliseconds":6.6241,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.001},{"name":"Read Collection","duration in milliseconds":6.4868000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:16:13.7114724Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"ba545b67-db8c-491c-9a9d-51e0acc43ca7","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.2656,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.2367,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:15:18.5596127Z","cpu":2.109,"memory":5235696.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0969,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:15:28.5689994Z","cpu":2.340,"memory":5235220.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.061,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:15:38.5748309Z","cpu":0.859,"memory":5234224.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.062,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:15:48.5902520Z","cpu":2.808,"memory":5233372.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.121,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:15:58.5925020Z","cpu":4.531,"memory":5192172.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.068,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:16:08.6033176Z","cpu":2.340,"memory":5205204.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0795,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.2308,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.2275,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.2087,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.2061,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.283,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:16:13.7271015Z","EndTimeUTC":"2026-04-14T18:16:13.7271015Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:16:13.7426759Z","DurationInMs":15.5744,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"e374f5a5-c260-432e-b409-5ee4811db155","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149714,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149714,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149714,"SessionToken":"-1#26149714","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.607","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:16:13 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:16:13.7271015Z", "durationInMs": 0.0097},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:16:13.7271112Z", "durationInMs": 17.4689},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:16:13.7445801Z", "durationInMs": 0.1855},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:16:13.7447656Z", "durationInMs": 1.769},{"event": "Received", "startTimeUtc": "2026-04-14T18:16:13.7465346Z", "durationInMs": 0.1249},{"event": "Completed", "startTimeUtc": "2026-04-14T18:16:13.7466595Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:16:13.7426759Z","lastSend":"2026-04-14T18:16:13.7426759Z","lastReceive":"2026-04-14T18:16:13.7426759Z"},"requestSizeInBytes":1635,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:16:13.7114724Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0705d1b8-fe34-4452-adcf-1f6d2db2e4b2","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:16:13.7114724Z","DurationInMs":15.629100000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"3bef7e49-3b91-423e-a759-a4480bb92c20","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:16:13.7271015Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"e374f5a5-c260-432e-b409-5ee4811db155","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1875075,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"e374f5a5-c260-432e-b409-5ee4811db155","ResponseTimeUtc":"2026-04-14T18:16:13.7426759Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: e374f5a5-c260-432e-b409-5ee4811db155, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149714","BELatencyInMs":"0.607"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 18:21:48 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: bca9ea38-7fe1-4b24-975a-74c77e5b2645; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: bca9ea38-7fe1-4b24-975a-74c77e5b2645, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: bca9ea38-7fe1-4b24-975a-74c77e5b2645; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: bca9ea38-7fe1-4b24-975a-74c77e5b2645, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:21:48.233Z","duration in milliseconds":49.793200000000006,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:21:47.5614970Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92231,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0391},{"name":"Get PkValue From Stream","duration in milliseconds":8.0022,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":7.8524,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:21:48.2333792Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"ecdf1d09-086f-4b0d-867e-738197cc8f9c","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":41.5244,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":41.4925,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:20:48.8303194Z","cpu":2.109,"memory":5219124.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1012,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:20:58.8445896Z","cpu":2.184,"memory":5219792.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0929,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:21:08.8586197Z","cpu":6.162,"memory":5247812.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0715,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:21:18.8652048Z","cpu":0.703,"memory":5244864.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1534,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:21:28.8740582Z","cpu":3.510,"memory":5239644.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0987,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:21:38.8897002Z","cpu":0.546,"memory":5236280.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1482,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":41.4846,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":41.4787,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":41.4538,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":41.451,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":40.2635,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:21:48.2490057Z","EndTimeUTC":"2026-04-14T18:21:48.2490057Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:21:48.2646321Z","DurationInMs":15.6264,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"bca9ea38-7fe1-4b24-975a-74c77e5b2645","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149736,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149736,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149736,"SessionToken":"-1#26149736","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.727","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:21:48 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:21:48.2490057Z", "durationInMs": 0.0077},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:21:48.2490134Z", "durationInMs": 18.8682},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:21:48.2678816Z", "durationInMs": 0.2114},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:21:48.2680930Z", "durationInMs": 1.9918},{"event": "Received", "startTimeUtc": "2026-04-14T18:21:48.2700848Z", "durationInMs": 0.142},{"event": "Completed", "startTimeUtc": "2026-04-14T18:21:48.2702268Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:21:48.2646321Z","lastSend":"2026-04-14T18:21:48.2646321Z","lastReceive":"2026-04-14T18:21:48.2646321Z"},"requestSizeInBytes":1622,"requestBodySizeInBytes":1172,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:21:48.2333792Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"42819d76-4df2-42ed-b260-eabad6c61fdd","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:21:48.2333792Z","DurationInMs":15.6265,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"fa5c0c66-ba6c-4ecb-992a-c50c03581951","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:21:48.2490057Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"bca9ea38-7fe1-4b24-975a-74c77e5b2645","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.7187572,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"bca9ea38-7fe1-4b24-975a-74c77e5b2645","ResponseTimeUtc":"2026-04-14T18:21:48.2802542Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: bca9ea38-7fe1-4b24-975a-74c77e5b2645, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149736","BELatencyInMs":"0.727"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0027}]}
2026-04-14 18:23:19 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b42f3ed5-b155-456b-a89c-e853500379fc; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b42f3ed5-b155-456b-a89c-e853500379fc, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b42f3ed5-b155-456b-a89c-e853500379fc; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b42f3ed5-b155-456b-a89c-e853500379fc, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:23:19.640Z","duration in milliseconds":46.029,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:23:19.5468416Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92249,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.054400000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":6.1485,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0015},{"name":"Read Collection","duration in milliseconds":5.9943,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:23:19.6405947Z","DurationInMs":15.622300000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"4e5f0d35-9a52-4df3-8880-8d18a039ff90","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":39.5899,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":39.5521,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:22:28.9263252Z","cpu":4.141,"memory":5250552.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1019,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:22:38.9393521Z","cpu":8.658,"memory":5218152.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1074,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:22:48.9491604Z","cpu":10.452,"memory":5215720.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0345,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:22:58.9596226Z","cpu":4.063,"memory":5318568.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0928,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:23:08.9758074Z","cpu":2.418,"memory":5302676.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0716,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:23:18.9844072Z","cpu":18.331,"memory":5216356.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0752,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":39.5415,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":39.5357,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":39.510400000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":39.507400000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":38.2564,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:23:19.6562170Z","EndTimeUTC":"2026-04-14T18:23:19.6562170Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:23:19.6874677Z","DurationInMs":31.250700000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"b42f3ed5-b155-456b-a89c-e853500379fc","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149736,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149736,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149736,"SessionToken":"-1#26149736","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.762","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:23:19 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:23:19.6562170Z", "durationInMs": 0.0082},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:23:19.6562252Z", "durationInMs": 22.0236},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:23:19.6782488Z", "durationInMs": 0.1978},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:23:19.6784466Z", "durationInMs": 2.11},{"event": "Received", "startTimeUtc": "2026-04-14T18:23:19.6805566Z", "durationInMs": 0.1504},{"event": "Completed", "startTimeUtc": "2026-04-14T18:23:19.6807070Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:23:19.6874677Z","lastSend":"2026-04-14T18:23:19.6874677Z","lastReceive":"2026-04-14T18:23:19.6874677Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1188,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:23:19.6562170Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"d8485b5a-1f62-4820-9875-0b91e249a55c","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:23:19.6562170Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"cda03b5a-16c7-4f94-a0f3-39ccdec47cc8","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:23:19.6562170Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"b42f3ed5-b155-456b-a89c-e853500379fc","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1406261,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"b42f3ed5-b155-456b-a89c-e853500379fc","ResponseTimeUtc":"2026-04-14T18:23:19.6874677Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: b42f3ed5-b155-456b-a89c-e853500379fc, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149736","BELatencyInMs":"0.762"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0024000000000000002}]}
2026-04-14 18:25:01 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 23438f7c-e7ed-4522-b7c7-f44b2cadb1db; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 23438f7c-e7ed-4522-b7c7-f44b2cadb1db, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 23438f7c-e7ed-4522-b7c7-f44b2cadb1db; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 23438f7c-e7ed-4522-b7c7-f44b2cadb1db, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:25:01.090Z","duration in milliseconds":44.935,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:25:01.0119063Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92303,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.057300000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":8.065100000000001,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0015},{"name":"Read Collection","duration in milliseconds":7.8848,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:25:01.0900318Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"560a1071-b619-4a40-9e36-27b255c31b64","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.5797,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.5323,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:24:09.0120635Z","cpu":6.552,"memory":5192100.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1007,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:24:19.0157135Z","cpu":1.406,"memory":5194368.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1005,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:24:29.0170064Z","cpu":5.469,"memory":5198228.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2198,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:24:39.0257348Z","cpu":7.266,"memory":5194236.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1344,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:24:49.0304437Z","cpu":4.446,"memory":5189196.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0503,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:24:59.0319890Z","cpu":8.203,"memory":5176300.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0841,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.522400000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.5174,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.4914,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.488600000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.399,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:25:01.1056622Z","EndTimeUTC":"2026-04-14T18:25:01.1056622Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:25:01.1369054Z","DurationInMs":31.2432,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"23438f7c-e7ed-4522-b7c7-f44b2cadb1db","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149856,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149856,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149856,"SessionToken":"-1#26149856","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.552","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:25:01 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:25:01.1056622Z", "durationInMs": 0.0086},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:25:01.1056708Z", "durationInMs": 18.6126},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:25:01.1242834Z", "durationInMs": 0.2002},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:25:01.1244836Z", "durationInMs": 1.9846},{"event": "Received", "startTimeUtc": "2026-04-14T18:25:01.1264682Z", "durationInMs": 0.1439},{"event": "Completed", "startTimeUtc": "2026-04-14T18:25:01.1266121Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:25:01.1212834Z","lastSend":"2026-04-14T18:25:01.1212834Z","lastReceive":"2026-04-14T18:25:01.1369054Z"},"requestSizeInBytes":1644,"requestBodySizeInBytes":1192,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:25:01.1056622Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"71f6eee4-35fd-4890-ab06-1334959e7c88","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:25:01.1056622Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"efbc31f2-9d0f-4486-a832-fac499392344","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:25:01.1056622Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"23438f7c-e7ed-4522-b7c7-f44b2cadb1db","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.12499909999999999,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"23438f7c-e7ed-4522-b7c7-f44b2cadb1db","ResponseTimeUtc":"2026-04-14T18:25:01.1369054Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 23438f7c-e7ed-4522-b7c7-f44b2cadb1db, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149856","BELatencyInMs":"0.552"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 18:27:14 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: ad26f000-0c45-4d59-ac6b-a97c1a73f02e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: ad26f000-0c45-4d59-ac6b-a97c1a73f02e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: ad26f000-0c45-4d59-ac6b-a97c1a73f02e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: ad26f000-0c45-4d59-ac6b-a97c1a73f02e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:27:14.633Z","duration in milliseconds":48.479,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:27:14.4771877Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92371,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.044000000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":6.5901000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0013000000000000002},{"name":"Read Collection","duration in milliseconds":6.4351,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:27:14.6335223Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"cddb0a7d-0e60-44ce-bee8-81cf6f904e09","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":41.6332,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":41.594500000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:26:19.1188672Z","cpu":15.211,"memory":5139844.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1143,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:26:29.1320783Z","cpu":3.978,"memory":5135628.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3635,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:26:39.1342848Z","cpu":3.203,"memory":5134956.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0826,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:26:49.1375781Z","cpu":2.578,"memory":5140996.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0916,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:26:59.1485580Z","cpu":7.566,"memory":5142692.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1035,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:27:09.1643849Z","cpu":5.616,"memory":5142412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1068,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":41.5858,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":41.5811,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":41.556200000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":41.552600000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":40.4117,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:27:14.6491301Z","EndTimeUTC":"2026-04-14T18:27:14.6491301Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:27:14.6803701Z","DurationInMs":31.240000000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"ad26f000-0c45-4d59-ac6b-a97c1a73f02e","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149858,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149858,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149858,"SessionToken":"-1#26149858","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.604","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:27:14 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:27:14.6491301Z", "durationInMs": 0.0092},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:27:14.6491393Z", "durationInMs": 22.4132},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:27:14.6715525Z", "durationInMs": 0.2108},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:27:14.6717633Z", "durationInMs": 2.9178},{"event": "Received", "startTimeUtc": "2026-04-14T18:27:14.6746811Z", "durationInMs": 0.1211},{"event": "Completed", "startTimeUtc": "2026-04-14T18:27:14.6748022Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:27:14.6646888Z","lastSend":"2026-04-14T18:27:14.6646888Z","lastReceive":"2026-04-14T18:27:14.6646888Z"},"requestSizeInBytes":1630,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:27:14.6335223Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0c2d695d-5da1-4288-b023-a23e3c8b9911","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:27:14.6335223Z","DurationInMs":15.607800000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"c305416a-37fc-42b5-8328-3433f3ba6699","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:27:14.6491301Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"ad26f000-0c45-4d59-ac6b-a97c1a73f02e","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.20318239999999999,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"ad26f000-0c45-4d59-ac6b-a97c1a73f02e","ResponseTimeUtc":"2026-04-14T18:27:14.6803701Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: ad26f000-0c45-4d59-ac6b-a97c1a73f02e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149858","BELatencyInMs":"0.604"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0033}]}
2026-04-14 18:28:09 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 85e62325-3416-4ae0-bcee-6db273ca3a23; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 85e62325-3416-4ae0-bcee-6db273ca3a23, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 85e62325-3416-4ae0-bcee-6db273ca3a23; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 85e62325-3416-4ae0-bcee-6db273ca3a23, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:28:09.705Z","duration in milliseconds":63.359100000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:28:09.5807023Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92401,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.050800000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":6.4872000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0036000000000000003},{"name":"Read Collection","duration in milliseconds":6.3269,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:28:09.7057507Z","DurationInMs":15.5776,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"c836e3d3-58d1-4a1a-8e89-ad2e166c6bd8","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":56.6218,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":56.584900000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:27:19.1756606Z","cpu":3.900,"memory":5142664.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1072,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:27:29.1785935Z","cpu":2.969,"memory":5143244.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0564,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:27:39.1852864Z","cpu":2.500,"memory":5142332.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0541,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:27:49.1929309Z","cpu":6.240,"memory":5151784.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0675,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:27:59.2065935Z","cpu":4.212,"memory":5250280.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0883,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:28:09.2214478Z","cpu":2.808,"memory":5233900.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0981,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":56.577200000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":56.571600000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":56.542100000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":56.5379,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":55.152300000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:28:09.7213283Z","EndTimeUTC":"2026-04-14T18:28:09.7369529Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:28:09.7682045Z","DurationInMs":31.2516,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"85e62325-3416-4ae0-bcee-6db273ca3a23","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149864,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149864,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149864,"SessionToken":"-1#26149864","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.703","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:28:09 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:28:09.7369529Z", "durationInMs": 0.0087},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:28:09.7369616Z", "durationInMs": 36.5222},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:28:09.7734838Z", "durationInMs": 0.2513},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:28:09.7737351Z", "durationInMs": 1.9644},{"event": "Received", "startTimeUtc": "2026-04-14T18:28:09.7756995Z", "durationInMs": 0.145},{"event": "Completed", "startTimeUtc": "2026-04-14T18:28:09.7758445Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:28:09.7682045Z","lastSend":"2026-04-14T18:28:09.7682045Z","lastReceive":"2026-04-14T18:28:09.7682045Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:28:09.7213283Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"b88180ec-89da-4a94-a7b5-f6a077c10c1c","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:28:09.7213283Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"bfe4fcd4-b91a-49c8-ae95-02fe0baf1be0","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:28:09.7213283Z","DurationInMs":15.624600000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"85e62325-3416-4ae0-bcee-6db273ca3a23","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.18750219999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"85e62325-3416-4ae0-bcee-6db273ca3a23","ResponseTimeUtc":"2026-04-14T18:28:09.7682045Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 85e62325-3416-4ae0-bcee-6db273ca3a23, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149864","BELatencyInMs":"0.703"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 18:28:28 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 2a7c857d-b866-4539-a8db-f37f6f2b34e5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 2a7c857d-b866-4539-a8db-f37f6f2b34e5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 2a7c857d-b866-4539-a8db-f37f6f2b34e5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 2a7c857d-b866-4539-a8db-f37f6f2b34e5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:28:28.486Z","duration in milliseconds":44.9549,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:28:28.3774878Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92412,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0451},{"name":"Get PkValue From Stream","duration in milliseconds":6.3222000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0043},{"name":"Read Collection","duration in milliseconds":6.1659000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:28:28.4868677Z","DurationInMs":15.6978,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"1e30b4fd-48a7-4bc6-b592-92a73c16b293","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":38.4003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":38.364000000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:27:29.1785935Z","cpu":2.969,"memory":5143244.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0564,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:27:39.1852864Z","cpu":2.500,"memory":5142332.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0541,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:27:49.1929309Z","cpu":6.240,"memory":5151784.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0675,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:27:59.2065935Z","cpu":4.212,"memory":5250280.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0883,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:28:09.2214478Z","cpu":2.808,"memory":5233900.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0981,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:28:19.2347367Z","cpu":4.141,"memory":5231352.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0921,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":38.3551,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":38.3498,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":38.3241,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":38.3215,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":37.3388,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:28:28.5025655Z","EndTimeUTC":"2026-04-14T18:28:28.5025655Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:28:28.5337436Z","DurationInMs":15.624500000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"2a7c857d-b866-4539-a8db-f37f6f2b34e5","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149864,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149864,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149864,"SessionToken":"-1#26149864","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.971","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:28:28 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:28:28.5181191Z", "durationInMs": 0.0117},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:28:28.5181308Z", "durationInMs": 19.7972},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:28:28.5379280Z", "durationInMs": 0.2202},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:28:28.5381482Z", "durationInMs": 2.3258},{"event": "Received", "startTimeUtc": "2026-04-14T18:28:28.5404740Z", "durationInMs": 0.1555},{"event": "Completed", "startTimeUtc": "2026-04-14T18:28:28.5406295Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:28:28.5337436Z","lastSend":"2026-04-14T18:28:28.5337436Z","lastReceive":"2026-04-14T18:28:28.5337436Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:28:28.5025655Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"e2984c1a-66ed-44f2-9000-bad42288dfcf","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:28:28.5025655Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"67f2d987-994a-46cb-8bc3-284ff2b3c480","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:28:28.5025655Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"2a7c857d-b866-4539-a8db-f37f6f2b34e5","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1562558,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"2a7c857d-b866-4539-a8db-f37f6f2b34e5","ResponseTimeUtc":"2026-04-14T18:28:28.5337436Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 2a7c857d-b866-4539-a8db-f37f6f2b34e5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149864","BELatencyInMs":"0.971"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 18:28:45 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 2024efa1-1a84-41eb-8512-65c6a1019139; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 2024efa1-1a84-41eb-8512-65c6a1019139, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 2024efa1-1a84-41eb-8512-65c6a1019139; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 2024efa1-1a84-41eb-8512-65c6a1019139, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:28:45.663Z","duration in milliseconds":44.153800000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:28:45.5535797Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92425,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0437},{"name":"Get PkValue From Stream","duration in milliseconds":6.4063,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0015},{"name":"Read Collection","duration in milliseconds":6.2661000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:28:45.6630141Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"ec6b2a38-9591-4260-8d23-cd2bb4df9a06","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.5334,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.5045,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:27:49.1929309Z","cpu":6.240,"memory":5151784.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0675,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:27:59.2065935Z","cpu":4.212,"memory":5250280.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0883,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:28:09.2214478Z","cpu":2.808,"memory":5233900.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0981,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:28:19.2347367Z","cpu":4.141,"memory":5231352.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0921,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:28:29.2426301Z","cpu":2.808,"memory":5249540.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0613,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:28:39.2536761Z","cpu":5.304,"memory":5246680.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.105,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.498000000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.4938,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.474000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.471900000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.6207,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:28:45.6786357Z","EndTimeUTC":"2026-04-14T18:28:45.6786357Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:28:45.7098294Z","DurationInMs":31.1937,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"2024efa1-1a84-41eb-8512-65c6a1019139","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149864,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149864,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149864,"SessionToken":"-1#26149864","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.698","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:28:45 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:28:45.6786357Z", "durationInMs": 0.0071},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:28:45.6786428Z", "durationInMs": 19.4431},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:28:45.6980859Z", "durationInMs": 0.2176},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:28:45.6983035Z", "durationInMs": 2.2015},{"event": "Received", "startTimeUtc": "2026-04-14T18:28:45.7005050Z", "durationInMs": 0.1453},{"event": "Completed", "startTimeUtc": "2026-04-14T18:28:45.7006503Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:28:45.6942043Z","lastSend":"2026-04-14T18:28:45.6942043Z","lastReceive":"2026-04-14T18:28:45.6942043Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:28:45.6630141Z","DurationInMs":15.6216,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"16596351-b71b-40a5-8c50-c6404cdbdd64","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:28:45.6786357Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"7f89dde6-6d82-41ed-af3a-f88bef87fa7d","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:28:45.6786357Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"2024efa1-1a84-41eb-8512-65c6a1019139","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1562497,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"2024efa1-1a84-41eb-8512-65c6a1019139","ResponseTimeUtc":"2026-04-14T18:28:45.7098294Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 2024efa1-1a84-41eb-8512-65c6a1019139, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149864","BELatencyInMs":"0.698"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0025}]}
2026-04-14 18:30:11 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5d234ac8-642f-45cf-b43c-11e94d0d21dc; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5d234ac8-642f-45cf-b43c-11e94d0d21dc, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5d234ac8-642f-45cf-b43c-11e94d0d21dc; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5d234ac8-642f-45cf-b43c-11e94d0d21dc, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:30:11.922Z","duration in milliseconds":60.691300000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:30:11.7503976Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92457,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0357},{"name":"Get PkValue From Stream","duration in milliseconds":6.354,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.2157,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:30:11.9223390Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"e1067188-ed56-44e4-9aa6-a19643ce0110","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":54.1345,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":54.1086,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:29:19.2758744Z","cpu":2.656,"memory":5218924.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.037,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:29:29.2869216Z","cpu":1.950,"memory":5215440.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1332,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:29:39.2954456Z","cpu":3.828,"memory":5211268.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0816,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:29:49.3029855Z","cpu":1.716,"memory":5213880.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0874,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:29:59.3165372Z","cpu":4.680,"memory":5212508.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.073,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:30:09.3244244Z","cpu":15.469,"memory":5152160.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0961,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":54.1017,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":54.098400000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":54.0799,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":54.077000000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":53.1424,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:30:11.9380516Z","EndTimeUTC":"2026-04-14T18:30:11.9380516Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:30:11.9692267Z","DurationInMs":31.1751,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"5d234ac8-642f-45cf-b43c-11e94d0d21dc","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149864,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149864,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149864,"SessionToken":"-1#26149864","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.879","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:30:11 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:30:11.9380516Z", "durationInMs": 0.0098},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:30:11.9380614Z", "durationInMs": 34.2934},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:30:11.9723548Z", "durationInMs": 0.211},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:30:11.9725658Z", "durationInMs": 2.0805},{"event": "Received", "startTimeUtc": "2026-04-14T18:30:11.9746463Z", "durationInMs": 0.1438},{"event": "Completed", "startTimeUtc": "2026-04-14T18:30:11.9747901Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:30:11.9692267Z","lastSend":"2026-04-14T18:30:11.9692267Z","lastReceive":"2026-04-14T18:30:11.9692267Z"},"requestSizeInBytes":1630,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:30:11.9223390Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"b4d575f9-e803-456b-8711-13d489e704ec","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:30:11.9223390Z","DurationInMs":15.7126,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"aaa3f938-7db5-4192-b6f9-f1e5e4643811","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:30:11.9380516Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"5d234ac8-642f-45cf-b43c-11e94d0d21dc","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2188291,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"5d234ac8-642f-45cf-b43c-11e94d0d21dc","ResponseTimeUtc":"2026-04-14T18:30:11.9692267Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 5d234ac8-642f-45cf-b43c-11e94d0d21dc, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149864","BELatencyInMs":"0.879"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 18:32:24 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 38b65134-63fe-4c6c-a119-670f711ff831; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 38b65134-63fe-4c6c-a119-670f711ff831, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 38b65134-63fe-4c6c-a119-670f711ff831; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 38b65134-63fe-4c6c-a119-670f711ff831, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:32:24.728Z","duration in milliseconds":42.5482,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:32:23.0874140Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92516,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0358},{"name":"Get PkValue From Stream","duration in milliseconds":6.2587,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":6.1161,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:32:24.7283410Z","DurationInMs":15.7009,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"7e7e6976-c23e-4f7c-a5b6-1cd58aaaa8b4","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.081700000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.0524,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:31:29.4041252Z","cpu":1.250,"memory":5161804.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1184,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:31:39.4090567Z","cpu":3.984,"memory":5162324.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0936,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:31:49.4107159Z","cpu":1.875,"memory":5162648.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0898,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:31:59.4245931Z","cpu":3.276,"memory":5163832.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1139,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:32:09.4329459Z","cpu":3.354,"memory":5162968.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0853,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:32:19.4400746Z","cpu":3.906,"memory":5170396.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0995,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.0464,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.0416,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.0217,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.019800000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.0681,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:32:24.7440419Z","EndTimeUTC":"2026-04-14T18:32:24.7440419Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:32:24.7752192Z","DurationInMs":15.6281,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"38b65134-63fe-4c6c-a119-670f711ff831","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149868,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149868,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149868,"SessionToken":"-1#26149868","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.595","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:32:24 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:32:24.7595911Z", "durationInMs": 0.0112},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:32:24.7596023Z", "durationInMs": 17.9399},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:32:24.7775422Z", "durationInMs": 0.2057},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:32:24.7777479Z", "durationInMs": 2.2916},{"event": "Received", "startTimeUtc": "2026-04-14T18:32:24.7800395Z", "durationInMs": 0.1402},{"event": "Completed", "startTimeUtc": "2026-04-14T18:32:24.7801797Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:32:24.7752192Z","lastSend":"2026-04-14T18:32:24.7752192Z","lastReceive":"2026-04-14T18:32:24.7752192Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1188,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:32:24.7440419Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"952396d2-3554-44c9-a028-93427c429b84","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:32:24.7440419Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"e8c90458-7f1c-4523-afe8-008fa617e3af","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:32:24.7440419Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"38b65134-63fe-4c6c-a119-670f711ff831","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":1.6878052,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"38b65134-63fe-4c6c-a119-670f711ff831","ResponseTimeUtc":"2026-04-14T18:32:24.7752192Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 38b65134-63fe-4c6c-a119-670f711ff831, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149868","BELatencyInMs":"0.595"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 18:33:23 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 941041fb-b95c-430a-9e84-a6fb9ff1b606; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 941041fb-b95c-430a-9e84-a6fb9ff1b606, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 941041fb-b95c-430a-9e84-a6fb9ff1b606; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 941041fb-b95c-430a-9e84-a6fb9ff1b606, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:33:23.390Z","duration in milliseconds":49.2573,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:33:23.2652607Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92545,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.047},{"name":"Get PkValue From Stream","duration in milliseconds":6.2277000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":6.0717,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:33:23.3902600Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"715f4360-ab06-4a49-a82c-a429a6c2fff0","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":42.8021,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":42.770500000000006,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:32:29.4562311Z","cpu":4.992,"memory":5182584.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0999,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:32:39.4696490Z","cpu":1.326,"memory":5181676.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1128,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:32:49.4820473Z","cpu":4.212,"memory":5180944.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3003,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:32:59.4842188Z","cpu":4.766,"memory":5343412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0975,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:33:09.4916241Z","cpu":3.354,"memory":5339288.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1055,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:33:19.4996598Z","cpu":3.359,"memory":5329068.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0875,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":42.760200000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":42.755500000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":42.736000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":42.732400000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":41.7261,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:33:23.4059437Z","EndTimeUTC":"2026-04-14T18:33:23.4059437Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:33:23.4371362Z","DurationInMs":31.192500000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"941041fb-b95c-430a-9e84-a6fb9ff1b606","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149868,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149868,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149868,"SessionToken":"-1#26149868","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.745","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:33:23 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:33:23.4059437Z", "durationInMs": 0.0083},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:33:23.4059520Z", "durationInMs": 24.8027},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:33:23.4307547Z", "durationInMs": 0.204},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:33:23.4309587Z", "durationInMs": 2.1802},{"event": "Received", "startTimeUtc": "2026-04-14T18:33:23.4331389Z", "durationInMs": 0.1483},{"event": "Completed", "startTimeUtc": "2026-04-14T18:33:23.4332872Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:33:23.4371362Z","lastSend":"2026-04-14T18:33:23.4371362Z","lastReceive":"2026-04-14T18:33:23.4371362Z"},"requestSizeInBytes":1628,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:33:23.3902600Z","DurationInMs":15.6837,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"eef0f430-631c-4c46-95d7-6db948dac8dd","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:33:23.4059437Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"8767b35f-07a1-49ac-9dd5-7e0c9e21b18c","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:33:23.4059437Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"941041fb-b95c-430a-9e84-a6fb9ff1b606","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718755,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"941041fb-b95c-430a-9e84-a6fb9ff1b606","ResponseTimeUtc":"2026-04-14T18:33:23.4371362Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 941041fb-b95c-430a-9e84-a6fb9ff1b606, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149868","BELatencyInMs":"0.745"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 18:33:51 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b3c84732-b0fc-4a4e-95f8-6c80562438b5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b3c84732-b0fc-4a4e-95f8-6c80562438b5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b3c84732-b0fc-4a4e-95f8-6c80562438b5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b3c84732-b0fc-4a4e-95f8-6c80562438b5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:33:51.004Z","duration in milliseconds":40.146,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:33:50.8798682Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92566,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0438},{"name":"Get PkValue From Stream","duration in milliseconds":6.0231,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":5.8874,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:33:51.0048737Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"8a5b76f9-539a-47d1-af4f-dbdebb4c209d","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":33.8598,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":33.833,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:32:59.4842188Z","cpu":4.766,"memory":5343412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0975,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:33:09.4916241Z","cpu":3.354,"memory":5339288.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1055,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:33:19.4996598Z","cpu":3.359,"memory":5329068.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0875,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:33:29.5060665Z","cpu":4.297,"memory":5324252.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2475,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:33:39.5213647Z","cpu":4.212,"memory":5321700.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0838,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1},{"dateUtc":"2026-04-14T18:33:49.5361677Z","cpu":2.496,"memory":5329176.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0396,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":33.8251,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":33.821000000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":33.798500000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":33.7947,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":32.4799,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:33:51.0204960Z","EndTimeUTC":"2026-04-14T18:33:51.0204960Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:33:51.0361330Z","DurationInMs":15.637,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"b3c84732-b0fc-4a4e-95f8-6c80562438b5","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149868,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149868,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149868,"SessionToken":"-1#26149868","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.528","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:33:51 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:33:51.0204960Z", "durationInMs": 0.0078},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:33:51.0205038Z", "durationInMs": 15.7729},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:33:51.0362767Z", "durationInMs": 0.1981},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:33:51.0364748Z", "durationInMs": 1.7904},{"event": "Received", "startTimeUtc": "2026-04-14T18:33:51.0382652Z", "durationInMs": 0.1338},{"event": "Completed", "startTimeUtc": "2026-04-14T18:33:51.0383990Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:33:51.0361330Z","lastSend":"2026-04-14T18:33:51.0361330Z","lastReceive":"2026-04-14T18:33:51.0361330Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:33:51.0048737Z","DurationInMs":15.622300000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"43422243-105d-4cdb-834b-f790a2cbed38","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:33:51.0204960Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"1390edd3-8b2e-4536-9382-214ea3d7323a","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:33:51.0204960Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"b3c84732-b0fc-4a4e-95f8-6c80562438b5","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.15626479999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"b3c84732-b0fc-4a4e-95f8-6c80562438b5","ResponseTimeUtc":"2026-04-14T18:33:51.0517466Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: b3c84732-b0fc-4a4e-95f8-6c80562438b5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149868","BELatencyInMs":"0.528"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 18:34:18 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e40c0cda-ae23-4abc-b59f-772500313ae0; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e40c0cda-ae23-4abc-b59f-772500313ae0, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e40c0cda-ae23-4abc-b59f-772500313ae0; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e40c0cda-ae23-4abc-b59f-772500313ae0, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:34:18.407Z","duration in milliseconds":44.1465,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:34:18.2827899Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92575,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.045700000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":6.7779000000000007,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.6332,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:34:18.4077931Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"68db4986-8ae8-43b3-abbc-a0a1268d2ede","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.147,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.1184,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:33:19.4996598Z","cpu":3.359,"memory":5329068.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0875,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:33:29.5060665Z","cpu":4.297,"memory":5324252.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2475,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:33:39.5213647Z","cpu":4.212,"memory":5321700.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0838,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1},{"dateUtc":"2026-04-14T18:33:49.5361677Z","cpu":2.496,"memory":5329176.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0396,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:33:59.5367573Z","cpu":5.313,"memory":5329568.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1239,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:09.5393592Z","cpu":4.602,"memory":5327052.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1044,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.1117,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.107800000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.086600000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.083800000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.140100000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:34:18.4234195Z","EndTimeUTC":"2026-04-14T18:34:18.4234195Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:34:18.4546651Z","DurationInMs":31.245600000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"e40c0cda-ae23-4abc-b59f-772500313ae0","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149871,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149871,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149871,"SessionToken":"-1#26149871","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.658","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:34:18 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:34:18.4234195Z", "durationInMs": 0.0072},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:34:18.4234267Z", "durationInMs": 20.3006},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:34:18.4437273Z", "durationInMs": 0.2106},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:34:18.4439379Z", "durationInMs": 1.745},{"event": "Received", "startTimeUtc": "2026-04-14T18:34:18.4456829Z", "durationInMs": 0.15},{"event": "Completed", "startTimeUtc": "2026-04-14T18:34:18.4458329Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:34:18.4391295Z","lastSend":"2026-04-14T18:34:18.4391295Z","lastReceive":"2026-04-14T18:34:18.4391295Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:34:18.4077931Z","DurationInMs":15.6264,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"f33eb479-df82-440b-84bc-4399080901e2","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:34:18.4234195Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0c2fef34-6cc2-4ef4-ac60-fd74c3574e4f","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:34:18.4234195Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"e40c0cda-ae23-4abc-b59f-772500313ae0","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718752,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"e40c0cda-ae23-4abc-b59f-772500313ae0","ResponseTimeUtc":"2026-04-14T18:34:18.4546651Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: e40c0cda-ae23-4abc-b59f-772500313ae0, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149871","BELatencyInMs":"0.658"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001}]}
2026-04-14 18:34:46 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d64d744b-1685-403d-ac19-3205c22a612b; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d64d744b-1685-403d-ac19-3205c22a612b, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d64d744b-1685-403d-ac19-3205c22a612b; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d64d744b-1685-403d-ac19-3205c22a612b, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:34:46.782Z","duration in milliseconds":41.805800000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:34:46.6575440Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92583,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0449},{"name":"Get PkValue From Stream","duration in milliseconds":5.9858,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0039000000000000003},{"name":"Read Collection","duration in milliseconds":5.8463,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:34:46.7825410Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"84b5f56a-3824-49fc-bcc3-5c826d1af795","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.5962,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.566700000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:33:49.5361677Z","cpu":2.496,"memory":5329176.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0396,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:33:59.5367573Z","cpu":5.313,"memory":5329568.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1239,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:09.5393592Z","cpu":4.602,"memory":5327052.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1044,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:19.5485200Z","cpu":2.813,"memory":5326572.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.084,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:29.5555603Z","cpu":2.496,"memory":5318392.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1203,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:39.5637164Z","cpu":3.516,"memory":5312124.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0995,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.5589,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.5548,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.5342,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.531600000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.2487,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:34:46.7982428Z","EndTimeUTC":"2026-04-14T18:34:46.7982428Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:34:46.8137936Z","DurationInMs":15.5508,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"d64d744b-1685-403d-ac19-3205c22a612b","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149872,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149872,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149872,"SessionToken":"-1#26149872","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.653","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:34:46 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:34:46.7982428Z", "durationInMs": 0.0075},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:34:46.7982503Z", "durationInMs": 17.0942},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:34:46.8153445Z", "durationInMs": 0.1772},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:34:46.8155217Z", "durationInMs": 2.0483},{"event": "Received", "startTimeUtc": "2026-04-14T18:34:46.8175700Z", "durationInMs": 0.144},{"event": "Completed", "startTimeUtc": "2026-04-14T18:34:46.8177140Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:34:46.8137936Z","lastSend":"2026-04-14T18:34:46.8137936Z","lastReceive":"2026-04-14T18:34:46.8137936Z"},"requestSizeInBytes":1630,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:34:46.7825410Z","DurationInMs":15.7018,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"6a9d0fe8-1e4b-4929-a32a-7ff5f33063b2","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:34:46.7982428Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"8700ef83-6c90-4510-9764-f1da01467b47","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:34:46.7982428Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"d64d744b-1685-403d-ac19-3205c22a612b","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1562496,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"d64d744b-1685-403d-ac19-3205c22a612b","ResponseTimeUtc":"2026-04-14T18:34:46.8294184Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: d64d744b-1685-403d-ac19-3205c22a612b, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149872","BELatencyInMs":"0.653"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 18:35:01 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 8e111d72-0cd1-4ad8-a239-7c10fe7f0a84; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 8e111d72-0cd1-4ad8-a239-7c10fe7f0a84, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 8e111d72-0cd1-4ad8-a239-7c10fe7f0a84; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 8e111d72-0cd1-4ad8-a239-7c10fe7f0a84, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:35:01.178Z","duration in milliseconds":90.4856,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:35:00.9914354Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92588,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0415},{"name":"Get PkValue From Stream","duration in milliseconds":6.0987,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":5.8290000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:35:01.1789368Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"e38773a7-0854-4902-a005-781e85769998","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":83.9512,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":83.89070000000001,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:34:09.5393592Z","cpu":4.602,"memory":5327052.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1044,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:19.5485200Z","cpu":2.813,"memory":5326572.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.084,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:29.5555603Z","cpu":2.496,"memory":5318392.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1203,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:39.5637164Z","cpu":3.516,"memory":5312124.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0995,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:49.5674559Z","cpu":2.656,"memory":5309872.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0905,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:59.5716441Z","cpu":15.757,"memory":5220044.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0901,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":83.8786,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":83.869,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":83.84190000000001,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":83.836700000000008,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":82.7436,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:35:01.2258161Z","EndTimeUTC":"2026-04-14T18:35:01.2258161Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:35:01.2726879Z","DurationInMs":46.8718,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"8e111d72-0cd1-4ad8-a239-7c10fe7f0a84","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149874,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149874,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149874,"SessionToken":"-1#26149874","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.578","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:35:01 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:35:01.2258161Z", "durationInMs": 0.0089},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:35:01.2258250Z", "durationInMs": 36.8962},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:35:01.2627212Z", "durationInMs": 0.2429},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:35:01.2629641Z", "durationInMs": 1.8961},{"event": "Received", "startTimeUtc": "2026-04-14T18:35:01.2648602Z", "durationInMs": 0.1769},{"event": "Completed", "startTimeUtc": "2026-04-14T18:35:01.2650371Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:35:01.2570646Z","lastSend":"2026-04-14T18:35:01.2570646Z","lastReceive":"2026-04-14T18:35:01.2570646Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:35:01.1789368Z","DurationInMs":15.627600000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"2fc0c633-5bf2-40cd-b6c0-5aab63501368","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:35:01.1945644Z","DurationInMs":31.251700000000003,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"92f2c1cf-64c0-4387-91c6-fdf9ccbc7496","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:35:01.2258161Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"8e111d72-0cd1-4ad8-a239-7c10fe7f0a84","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.28125249999999996,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"8e111d72-0cd1-4ad8-a239-7c10fe7f0a84","ResponseTimeUtc":"2026-04-14T18:35:01.2726879Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 8e111d72-0cd1-4ad8-a239-7c10fe7f0a84, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149874","BELatencyInMs":"0.578"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 18:35:10 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 3b4758b9-175c-4274-b3db-18b238aa60c4; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 3b4758b9-175c-4274-b3db-18b238aa60c4, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 3b4758b9-175c-4274-b3db-18b238aa60c4; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 3b4758b9-175c-4274-b3db-18b238aa60c4, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:35:10.331Z","duration in milliseconds":61.815900000000006,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:35:10.2060427Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92591,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0361},{"name":"Get PkValue From Stream","duration in milliseconds":6.9742000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.00090000000000000008},{"name":"Read Collection","duration in milliseconds":6.6764,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:35:10.3310451Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"006dfc81-9cc2-47ad-994e-2d22fb966736","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":54.3814,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":54.3451,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:34:19.5485200Z","cpu":2.813,"memory":5326572.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.084,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:29.5555603Z","cpu":2.496,"memory":5318392.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1203,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:39.5637164Z","cpu":3.516,"memory":5312124.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0995,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:49.5674559Z","cpu":2.656,"memory":5309872.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0905,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:59.5716441Z","cpu":15.757,"memory":5220044.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0901,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:35:09.5926159Z","cpu":6.162,"memory":5195952.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.125,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":54.3316,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":54.3224,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":54.2971,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":54.292,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":53.1051,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:35:10.3466703Z","EndTimeUTC":"2026-04-14T18:35:10.3466703Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:35:10.3803334Z","DurationInMs":33.6631,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"3b4758b9-175c-4274-b3db-18b238aa60c4","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149874,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149874,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149874,"SessionToken":"-1#26149874","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"1.013","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:35:10 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:35:10.3466703Z", "durationInMs": 0.0071},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:35:10.3466774Z", "durationInMs": 36.395},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:35:10.3830724Z", "durationInMs": 0.3279},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:35:10.3834003Z", "durationInMs": 2.1722},{"event": "Received", "startTimeUtc": "2026-04-14T18:35:10.3855725Z", "durationInMs": 0.1808},{"event": "Completed", "startTimeUtc": "2026-04-14T18:35:10.3857533Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:35:10.3622963Z","lastSend":"2026-04-14T18:35:10.3622963Z","lastReceive":"2026-04-14T18:35:10.3803334Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:35:10.3310451Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"54d16f8a-ad52-444d-8ce0-77a53ded23f8","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:35:10.3310451Z","DurationInMs":15.625200000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"96f121c5-405b-4871-9959-fc4573eb3ef5","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:35:10.3466703Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"3b4758b9-175c-4274-b3db-18b238aa60c4","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1742907,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"3b4758b9-175c-4274-b3db-18b238aa60c4","ResponseTimeUtc":"2026-04-14T18:35:10.3803334Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 3b4758b9-175c-4274-b3db-18b238aa60c4, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149874","BELatencyInMs":"1.013"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0031000000000000003}]}
2026-04-14 18:35:31 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 094d9590-6bfe-40eb-a123-d665028fd825; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 094d9590-6bfe-40eb-a123-d665028fd825, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 094d9590-6bfe-40eb-a123-d665028fd825; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 094d9590-6bfe-40eb-a123-d665028fd825, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:35:31.484Z","duration in milliseconds":45.6148,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:35:31.3593808Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92597,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.037700000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":7.9835,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.001},{"name":"Read Collection","duration in milliseconds":7.7828,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:35:31.4843842Z","DurationInMs":15.6225,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"f5de1855-5e87-4694-8779-b813121fc44e","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.3847,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.348400000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:34:39.5637164Z","cpu":3.516,"memory":5312124.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0995,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:49.5674559Z","cpu":2.656,"memory":5309872.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0905,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:59.5716441Z","cpu":15.757,"memory":5220044.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0901,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:35:09.5926159Z","cpu":6.162,"memory":5195952.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.125,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:35:19.5950757Z","cpu":3.438,"memory":5189288.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1688,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:35:29.6093446Z","cpu":5.694,"memory":5232600.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1753,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.341,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.3363,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.3112,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.3076,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.094300000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:35:31.5000067Z","EndTimeUTC":"2026-04-14T18:35:31.5000067Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:35:31.5312571Z","DurationInMs":31.250400000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"094d9590-6bfe-40eb-a123-d665028fd825","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149875,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149875,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149875,"SessionToken":"-1#26149875","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.805","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:35:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:35:31.5000067Z", "durationInMs": 0.0114},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:35:31.5000181Z", "durationInMs": 18.7712},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:35:31.5187893Z", "durationInMs": 0.1733},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:35:31.5189626Z", "durationInMs": 2.4275},{"event": "Received", "startTimeUtc": "2026-04-14T18:35:31.5213901Z", "durationInMs": 0.1113},{"event": "Completed", "startTimeUtc": "2026-04-14T18:35:31.5215014Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:35:31.5156341Z","lastSend":"2026-04-14T18:35:31.5156341Z","lastReceive":"2026-04-14T18:35:31.5312571Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:35:31.5000067Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"f925aa3f-818c-4d73-973b-97e63add6d20","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:35:31.5000067Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"9a9b38f6-afe6-42ba-b64e-e9f8a3d55a6e","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:35:31.5000067Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"094d9590-6bfe-40eb-a123-d665028fd825","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.17187629999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"094d9590-6bfe-40eb-a123-d665028fd825","ResponseTimeUtc":"2026-04-14T18:35:31.5312571Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 094d9590-6bfe-40eb-a123-d665028fd825, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149875","BELatencyInMs":"0.805"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 18:35:48 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 9adaa31b-3abb-402b-a270-093ffb26075e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 9adaa31b-3abb-402b-a270-093ffb26075e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 9adaa31b-3abb-402b-a270-093ffb26075e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 9adaa31b-3abb-402b-a270-093ffb26075e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:35:48.944Z","duration in milliseconds":46.8226,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:35:48.8190444Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92603,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0369},{"name":"Get PkValue From Stream","duration in milliseconds":7.5983,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":7.4167000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:35:48.9440455Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"0e7b4078-dc69-4d2b-8404-3ff894f559dd","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":38.7973,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":38.7569,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:34:49.5674559Z","cpu":2.656,"memory":5309872.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0905,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:34:59.5716441Z","cpu":15.757,"memory":5220044.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0901,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:35:09.5926159Z","cpu":6.162,"memory":5195952.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.125,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:35:19.5950757Z","cpu":3.438,"memory":5189288.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1688,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:35:29.6093446Z","cpu":5.694,"memory":5232600.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1753,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:35:39.6222791Z","cpu":16.225,"memory":5232960.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1606,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":38.7455,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":38.7357,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":38.7078,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":38.7029,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":37.6653,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:35:48.9596692Z","EndTimeUTC":"2026-04-14T18:35:48.9596692Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:35:48.9909225Z","DurationInMs":31.253300000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"9adaa31b-3abb-402b-a270-093ffb26075e","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149876,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149876,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149876,"SessionToken":"-1#26149876","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.578","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:35:48 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:35:48.9596692Z", "durationInMs": 0.0077},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:35:48.9596769Z", "durationInMs": 19.0473},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:35:48.9787242Z", "durationInMs": 0.2566},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:35:48.9789808Z", "durationInMs": 2.0763},{"event": "Received", "startTimeUtc": "2026-04-14T18:35:48.9810571Z", "durationInMs": 0.1752},{"event": "Completed", "startTimeUtc": "2026-04-14T18:35:48.9812323Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:35:48.9753883Z","lastSend":"2026-04-14T18:35:48.9753883Z","lastReceive":"2026-04-14T18:35:48.9753883Z"},"requestSizeInBytes":1628,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:35:48.9440455Z","DurationInMs":15.623700000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"40c02a7c-696d-4681-a34c-38a7519eefd9","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:35:48.9596692Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"9573e645-8e3a-488b-9876-80ab74995915","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:35:48.9596692Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"9adaa31b-3abb-402b-a270-093ffb26075e","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718781,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"9adaa31b-3abb-402b-a270-093ffb26075e","ResponseTimeUtc":"2026-04-14T18:35:48.9909225Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 9adaa31b-3abb-402b-a270-093ffb26075e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149876","BELatencyInMs":"0.578"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0013000000000000002}]}
2026-04-14 18:37:22 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: fd884226-66c2-40e2-af88-e495d078e25b; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: fd884226-66c2-40e2-af88-e495d078e25b, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: fd884226-66c2-40e2-af88-e495d078e25b; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: fd884226-66c2-40e2-af88-e495d078e25b, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:37:22.034Z","duration in milliseconds":46.909200000000006,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:37:21.9095132Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92643,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0436},{"name":"Get PkValue From Stream","duration in milliseconds":7.2087,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0013000000000000002},{"name":"Read Collection","duration in milliseconds":7.0711,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:37:22.0346088Z","DurationInMs":15.603000000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"7784d057-4fab-45e8-9729-0ad95a6caa5d","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":39.4846,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":39.456900000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:36:29.6752865Z","cpu":3.432,"memory":5170160.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1215,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:36:39.6789733Z","cpu":0.859,"memory":5166572.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1054,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:36:49.6853198Z","cpu":2.886,"memory":5158248.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0971,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:36:59.7008136Z","cpu":4.446,"memory":5150228.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1012,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:09.7022403Z","cpu":4.688,"memory":5151712.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1308,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:19.7168075Z","cpu":5.226,"memory":5152304.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1024,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":39.4499,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":39.4459,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":39.426500000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":39.4243,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":38.4893,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:37:22.0502118Z","EndTimeUTC":"2026-04-14T18:37:22.0658252Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:37:22.0813916Z","DurationInMs":15.566400000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"fd884226-66c2-40e2-af88-e495d078e25b","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149882,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149882,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149882,"SessionToken":"-1#26149882","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.656","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:37:22 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:37:22.0658252Z", "durationInMs": 0.0103},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:37:22.0658355Z", "durationInMs": 21.4968},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:37:22.0873323Z", "durationInMs": 0.1958},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:37:22.0875281Z", "durationInMs": 1.835},{"event": "Received", "startTimeUtc": "2026-04-14T18:37:22.0893631Z", "durationInMs": 0.1427},{"event": "Completed", "startTimeUtc": "2026-04-14T18:37:22.0895058Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:37:22.0813916Z","lastSend":"2026-04-14T18:37:22.0813916Z","lastReceive":"2026-04-14T18:37:22.0813916Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:37:22.0502118Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"fe529e7e-0e1c-403b-a75a-68a5414bf491","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:37:22.0502118Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"ecdf7125-7081-4d3a-91c6-8dbaf069525d","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:37:22.0502118Z","DurationInMs":15.6134,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"fd884226-66c2-40e2-af88-e495d078e25b","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718784,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"fd884226-66c2-40e2-af88-e495d078e25b","ResponseTimeUtc":"2026-04-14T18:37:22.0813916Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: fd884226-66c2-40e2-af88-e495d078e25b, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149882","BELatencyInMs":"0.656"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 18:37:37 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 40fc3286-00e4-408d-b170-5034d8aaf19f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 40fc3286-00e4-408d-b170-5034d8aaf19f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 40fc3286-00e4-408d-b170-5034d8aaf19f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 40fc3286-00e4-408d-b170-5034d8aaf19f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:37:37.279Z","duration in milliseconds":42.469,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:37:37.1542029Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92663,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0562},{"name":"Get PkValue From Stream","duration in milliseconds":7.2471000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0013000000000000002},{"name":"Read Collection","duration in milliseconds":7.0227,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:37:37.2792667Z","DurationInMs":15.5717,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"bffce8f2-6d85-4707-86d6-380a800bd715","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":34.9742,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":34.9303,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:36:39.6789733Z","cpu":0.859,"memory":5166572.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1054,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:36:49.6853198Z","cpu":2.886,"memory":5158248.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0971,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:36:59.7008136Z","cpu":4.446,"memory":5150228.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1012,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:09.7022403Z","cpu":4.688,"memory":5151712.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1308,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:19.7168075Z","cpu":5.226,"memory":5152304.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1024,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:29.7194909Z","cpu":5.547,"memory":5155568.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0937,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":34.9232,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":34.9194,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":34.8968,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":34.894200000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":33.9057,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:37:37.2948384Z","EndTimeUTC":"2026-04-14T18:37:37.3105171Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:37:37.3260731Z","DurationInMs":15.556000000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"40fc3286-00e4-408d-b170-5034d8aaf19f","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149882,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149882,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149882,"SessionToken":"-1#26149882","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.636","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:37:37 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:37:37.3105171Z", "durationInMs": 0.0115},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:37:37.3105286Z", "durationInMs": 17.4598},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:37:37.3279884Z", "durationInMs": 0.2015},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:37:37.3281899Z", "durationInMs": 1.9781},{"event": "Received", "startTimeUtc": "2026-04-14T18:37:37.3301680Z", "durationInMs": 0.1347},{"event": "Completed", "startTimeUtc": "2026-04-14T18:37:37.3303027Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:37:37.3105171Z","lastSend":"2026-04-14T18:37:37.3105171Z","lastReceive":"2026-04-14T18:37:37.3260731Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:37:37.2948384Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"d868992a-d1e9-45fe-9c81-7e56109da710","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:37:37.2948384Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"d9329fec-5656-48bc-ac4c-d388bd4a38a9","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:37:37.2948384Z","DurationInMs":15.678700000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"40fc3286-00e4-408d-b170-5034d8aaf19f","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718702,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"40fc3286-00e4-408d-b170-5034d8aaf19f","ResponseTimeUtc":"2026-04-14T18:37:37.3260731Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 40fc3286-00e4-408d-b170-5034d8aaf19f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149882","BELatencyInMs":"0.636"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0029000000000000002}]}
2026-04-14 18:37:51 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 3c955829-5344-4959-a111-d993004b40ad; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 3c955829-5344-4959-a111-d993004b40ad, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 3c955829-5344-4959-a111-d993004b40ad; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 3c955829-5344-4959-a111-d993004b40ad, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:37:51.306Z","duration in milliseconds":38.6357,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:37:51.1969024Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92667,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.036500000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":6.4622,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0038},{"name":"Read Collection","duration in milliseconds":6.3245000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:37:51.3062816Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"6881fe2d-7955-4ece-8409-0cd898d3f13b","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":31.971300000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":31.944000000000003,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:36:59.7008136Z","cpu":4.446,"memory":5150228.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1012,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:09.7022403Z","cpu":4.688,"memory":5151712.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1308,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:19.7168075Z","cpu":5.226,"memory":5152304.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1024,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:29.7194909Z","cpu":5.547,"memory":5155568.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0937,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:39.7324675Z","cpu":4.602,"memory":5153920.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.132,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:49.7338938Z","cpu":2.266,"memory":5159908.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1071,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":31.937900000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":31.9338,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":31.9162,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":31.9141,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":31.0786,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:37:51.3219052Z","EndTimeUTC":"2026-04-14T18:37:51.3219052Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:37:51.3375306Z","DurationInMs":15.6254,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"3c955829-5344-4959-a111-d993004b40ad","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149883,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149883,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149883,"SessionToken":"-1#26149883","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.534","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:37:51 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:37:51.3219052Z", "durationInMs": 0.0067},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:37:51.3219119Z", "durationInMs": 15.35},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:37:51.3372619Z", "durationInMs": 0.2032},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:37:51.3374651Z", "durationInMs": 1.7064},{"event": "Received", "startTimeUtc": "2026-04-14T18:37:51.3391715Z", "durationInMs": 0.1404},{"event": "Completed", "startTimeUtc": "2026-04-14T18:37:51.3393119Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:37:51.3375306Z","lastSend":"2026-04-14T18:37:51.3375306Z","lastReceive":"2026-04-14T18:37:51.3375306Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1186,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:37:51.3062816Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"3796ed4f-621d-464a-b176-23d33f331ff9","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:37:51.3062816Z","DurationInMs":15.623600000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"3849ad24-9b17-44fc-8ffb-a0e2d96ba6af","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:37:51.3219052Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"3c955829-5344-4959-a111-d993004b40ad","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.14062819999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"3c955829-5344-4959-a111-d993004b40ad","ResponseTimeUtc":"2026-04-14T18:37:51.3375306Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 3c955829-5344-4959-a111-d993004b40ad, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149883","BELatencyInMs":"0.534"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 18:38:05 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6da3e5ff-de2c-48c2-ae66-f65da634db8e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6da3e5ff-de2c-48c2-ae66-f65da634db8e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6da3e5ff-de2c-48c2-ae66-f65da634db8e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6da3e5ff-de2c-48c2-ae66-f65da634db8e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:38:05.859Z","duration in milliseconds":43.8874,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:38:05.7499926Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92670,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0437},{"name":"Get PkValue From Stream","duration in milliseconds":6.8135,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.6676,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:38:05.8749945Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"9b0b9d8c-996a-44fc-8d57-d7fd6236ba87","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.8639,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.832,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:37:09.7022403Z","cpu":4.688,"memory":5151712.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1308,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:19.7168075Z","cpu":5.226,"memory":5152304.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1024,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:29.7194909Z","cpu":5.547,"memory":5155568.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0937,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:39.7324675Z","cpu":4.602,"memory":5153920.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.132,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:49.7338938Z","cpu":2.266,"memory":5159908.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1071,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:59.7453660Z","cpu":3.281,"memory":5259768.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0955,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.8234,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.8189,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.798700000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.7959,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.9557,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:38:05.8907219Z","EndTimeUTC":"2026-04-14T18:38:05.8907219Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:38:05.9062465Z","DurationInMs":15.524600000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"6da3e5ff-de2c-48c2-ae66-f65da634db8e","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149883,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149883,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149883,"SessionToken":"-1#26149883","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.622","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:38:05 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:38:05.8907219Z", "durationInMs": 0.0104},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:38:05.8907323Z", "durationInMs": 18.2724},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:38:05.9090047Z", "durationInMs": 0.2013},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:38:05.9092060Z", "durationInMs": 2.2301},{"event": "Received", "startTimeUtc": "2026-04-14T18:38:05.9114361Z", "durationInMs": 0.1492},{"event": "Completed", "startTimeUtc": "2026-04-14T18:38:05.9115853Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:38:05.9062465Z","lastSend":"2026-04-14T18:38:05.9062465Z","lastReceive":"2026-04-14T18:38:05.9062465Z"},"requestSizeInBytes":1638,"requestBodySizeInBytes":1186,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:38:05.8749945Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"4c01f597-2163-4126-b584-913e58c26756","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:38:05.8749945Z","DurationInMs":15.727400000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"38748420-92d2-4bcd-b4f8-1274967ab13c","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:38:05.8907219Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"6da3e5ff-de2c-48c2-ae66-f65da634db8e","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1562539,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"6da3e5ff-de2c-48c2-ae66-f65da634db8e","ResponseTimeUtc":"2026-04-14T18:38:05.9062465Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 6da3e5ff-de2c-48c2-ae66-f65da634db8e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149883","BELatencyInMs":"0.622"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 18:38:17 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: eecc88b2-f7d0-4054-a5b2-7bdde85de1d1; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: eecc88b2-f7d0-4054-a5b2-7bdde85de1d1, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: eecc88b2-f7d0-4054-a5b2-7bdde85de1d1; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: eecc88b2-f7d0-4054-a5b2-7bdde85de1d1, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:38:17.333Z","duration in milliseconds":47.24,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:38:17.2552356Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92673,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.049300000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":6.7992,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.002},{"name":"Read Collection","duration in milliseconds":6.6293000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:38:17.3333619Z","DurationInMs":15.625100000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"0be39ca9-dada-44d9-b99d-b8011fce97a3","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":40.209700000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":40.1709,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:37:19.7168075Z","cpu":5.226,"memory":5152304.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1024,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:29.7194909Z","cpu":5.547,"memory":5155568.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0937,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:39.7324675Z","cpu":4.602,"memory":5153920.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.132,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:49.7338938Z","cpu":2.266,"memory":5159908.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1071,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:37:59.7453660Z","cpu":3.281,"memory":5259768.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0955,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:38:09.7565379Z","cpu":1.638,"memory":5253468.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1033,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":40.163900000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":40.1588,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":40.139700000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":40.1363,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":39.272600000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:38:17.3646683Z","EndTimeUTC":"2026-04-14T18:38:17.3646683Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:38:17.3802400Z","DurationInMs":15.5717,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"eecc88b2-f7d0-4054-a5b2-7bdde85de1d1","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149883,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149883,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149883,"SessionToken":"-1#26149883","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.559","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:38:17 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:38:17.3646683Z", "durationInMs": 0.0087},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:38:17.3646770Z", "durationInMs": 21.6193},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:38:17.3862963Z", "durationInMs": 0.2036},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:38:17.3864999Z", "durationInMs": 2.0365},{"event": "Received", "startTimeUtc": "2026-04-14T18:38:17.3885364Z", "durationInMs": 0.1471},{"event": "Completed", "startTimeUtc": "2026-04-14T18:38:17.3886835Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:38:17.3802400Z","lastSend":"2026-04-14T18:38:17.3802400Z","lastReceive":"2026-04-14T18:38:17.3802400Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1186,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:38:17.3489870Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"570c38f4-d160-4950-902a-c1d25a433b9f","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:38:17.3489870Z","DurationInMs":15.6813,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"67604883-ee46-472d-8430-25e3f2e1626b","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:38:17.3646683Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"eecc88b2-f7d0-4054-a5b2-7bdde85de1d1","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.140628,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"eecc88b2-f7d0-4054-a5b2-7bdde85de1d1","ResponseTimeUtc":"2026-04-14T18:38:17.3802400Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: eecc88b2-f7d0-4054-a5b2-7bdde85de1d1, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149883","BELatencyInMs":"0.559"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0015}]}
2026-04-14 18:39:26 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 69ebb3fe-d728-4cf2-b27e-375759b2da44; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 69ebb3fe-d728-4cf2-b27e-375759b2da44, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 69ebb3fe-d728-4cf2-b27e-375759b2da44; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 69ebb3fe-d728-4cf2-b27e-375759b2da44, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:39:25.968Z","duration in milliseconds":50.1976,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:39:25.4998097Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92679,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.037200000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":8.3301,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0036000000000000003},{"name":"Read Collection","duration in milliseconds":8.1794000000000011,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:39:25.9686107Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"8b7c7c51-fffb-4500-afba-b021b917605b","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":41.659400000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":41.630300000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:38:29.7758836Z","cpu":1.953,"memory":5271684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1015,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:38:39.7914626Z","cpu":2.496,"memory":5269400.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1155,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:38:49.7976722Z","cpu":1.014,"memory":5270512.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1147,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:38:59.8112702Z","cpu":1.170,"memory":5267640.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1169,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:39:09.8259707Z","cpu":3.588,"memory":5267512.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1323,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:39:19.8294699Z","cpu":1.094,"memory":5259628.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1179,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":41.621500000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":41.6176,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":41.5955,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":41.5934,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":40.6405,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:39:25.9842602Z","EndTimeUTC":"2026-04-14T18:39:25.9842602Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:39:26.0154433Z","DurationInMs":31.183100000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"69ebb3fe-d728-4cf2-b27e-375759b2da44","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149883,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149883,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149883,"SessionToken":"-1#26149883","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.633","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:39:26 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:39:25.9842602Z", "durationInMs": 0.0084},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:39:25.9842686Z", "durationInMs": 24.1126},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:39:26.0083812Z", "durationInMs": 0.1935},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:39:26.0085747Z", "durationInMs": 2.0397},{"event": "Received", "startTimeUtc": "2026-04-14T18:39:26.0106144Z", "durationInMs": 0.1397},{"event": "Completed", "startTimeUtc": "2026-04-14T18:39:26.0107541Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:39:25.9998163Z","lastSend":"2026-04-14T18:39:25.9998163Z","lastReceive":"2026-04-14T18:39:26.0154433Z"},"requestSizeInBytes":1623,"requestBodySizeInBytes":1173,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:39:25.9686107Z","DurationInMs":15.649500000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"21f476d0-c03c-4054-99bb-67ea31760e0e","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:39:25.9842602Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"b13e0887-4ba7-4fbf-92ac-e389149e01c9","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:39:25.9842602Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"69ebb3fe-d728-4cf2-b27e-375759b2da44","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.5156336,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"69ebb3fe-d728-4cf2-b27e-375759b2da44","ResponseTimeUtc":"2026-04-14T18:39:26.0154433Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 69ebb3fe-d728-4cf2-b27e-375759b2da44, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149883","BELatencyInMs":"0.633"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001}]}
2026-04-14 18:44:33 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 04fede4d-27b9-4f43-8a56-ab3e9752395e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 04fede4d-27b9-4f43-8a56-ab3e9752395e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 04fede4d-27b9-4f43-8a56-ab3e9752395e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 04fede4d-27b9-4f43-8a56-ab3e9752395e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:44:33.269Z","duration in milliseconds":44.748400000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:44:33.1755048Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92754,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.057},{"name":"Get PkValue From Stream","duration in milliseconds":6.8333,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002},{"name":"Read Collection","duration in milliseconds":6.617,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:44:33.2692542Z","DurationInMs":15.629000000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"3b434685-172e-4e43-a001-77b621720622","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.454,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.4061,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:43:40.0866182Z","cpu":11.232,"memory":5198084.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0415,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:43:50.0978375Z","cpu":3.588,"memory":5202224.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1144,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:44:00.1125265Z","cpu":2.418,"memory":5202496.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0317,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:44:10.1178420Z","cpu":1.016,"memory":5200900.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0707,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:44:20.1252556Z","cpu":2.344,"memory":5191412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1194,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:44:30.1298894Z","cpu":12.949,"memory":5149588.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1181,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.3933,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.3842,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.3572,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.3522,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.1501,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:44:33.2848832Z","EndTimeUTC":"2026-04-14T18:44:33.3005041Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:44:33.3161282Z","DurationInMs":15.6241,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"04fede4d-27b9-4f43-8a56-ab3e9752395e","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149887,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149887,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149887,"SessionToken":"-1#26149887","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.754","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:44:33 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:44:33.3005041Z", "durationInMs": 0.0084},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:44:33.3005125Z", "durationInMs": 17.7183},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:44:33.3182308Z", "durationInMs": 0.231},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:44:33.3184618Z", "durationInMs": 2.9868},{"event": "Received", "startTimeUtc": "2026-04-14T18:44:33.3214486Z", "durationInMs": 0.1662},{"event": "Completed", "startTimeUtc": "2026-04-14T18:44:33.3216148Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:44:33.3161282Z","lastSend":"2026-04-14T18:44:33.3161282Z","lastReceive":"2026-04-14T18:44:33.3161282Z"},"requestSizeInBytes":1637,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:44:33.2848832Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"b5d2db15-49af-4ecf-baba-74ef282c76b5","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:44:33.2848832Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"5c4a4907-dc73-446b-bc10-05f2f85c8d49","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:44:33.2848832Z","DurationInMs":15.6209,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"04fede4d-27b9-4f43-8a56-ab3e9752395e","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.14062339999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"04fede4d-27b9-4f43-8a56-ab3e9752395e","ResponseTimeUtc":"2026-04-14T18:44:33.3161282Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 04fede4d-27b9-4f43-8a56-ab3e9752395e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149887","BELatencyInMs":"0.754"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 18:45:25 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6702dc55-72da-4936-87a5-0b476bb48d6f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6702dc55-72da-4936-87a5-0b476bb48d6f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6702dc55-72da-4936-87a5-0b476bb48d6f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6702dc55-72da-4936-87a5-0b476bb48d6f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:45:25.934Z","duration in milliseconds":52.068200000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:45:25.8412002Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92763,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0392},{"name":"Get PkValue From Stream","duration in milliseconds":6.4715000000000007,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0033},{"name":"Read Collection","duration in milliseconds":6.28,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:45:25.9349476Z","DurationInMs":15.624400000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"37ffa155-9ce8-4972-afcf-945259a38381","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":45.1227,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":45.081500000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:44:30.1298894Z","cpu":12.949,"memory":5149588.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1181,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:44:40.1339485Z","cpu":3.203,"memory":5144456.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1528,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:44:50.1387669Z","cpu":2.031,"memory":5137492.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1054,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:45:00.1437077Z","cpu":1.094,"memory":5136340.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":2.3929,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:45:10.1458115Z","cpu":28.627,"memory":5123440.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1341,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:45:20.1471954Z","cpu":10.391,"memory":5093228.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0616,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":45.067800000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":45.058800000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":45.0219,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":45.0143,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":43.761700000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:45:25.9505720Z","EndTimeUTC":"2026-04-14T18:45:25.9661981Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:45:25.9857811Z","DurationInMs":19.583000000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"6702dc55-72da-4936-87a5-0b476bb48d6f","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149888,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149888,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149888,"SessionToken":"-1#26149888","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.709","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:45:25 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:45:25.9661981Z", "durationInMs": 0.0078},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:45:25.9662059Z", "durationInMs": 24.8286},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:45:25.9910345Z", "durationInMs": 0.2383},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:45:25.9912728Z", "durationInMs": 2.364},{"event": "Received", "startTimeUtc": "2026-04-14T18:45:25.9936368Z", "durationInMs": 0.2024},{"event": "Completed", "startTimeUtc": "2026-04-14T18:45:25.9938392Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:45:25.9857811Z","lastSend":"2026-04-14T18:45:25.9857811Z","lastReceive":"2026-04-14T18:45:25.9857811Z"},"requestSizeInBytes":1638,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:45:25.9505720Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"3d5b7de3-372c-457e-8136-f696a753453b","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:45:25.9505720Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"117c246d-cfad-4576-ae36-dbfca53e388d","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:45:25.9505720Z","DurationInMs":15.626100000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"6702dc55-72da-4936-87a5-0b476bb48d6f","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.14458089999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"6702dc55-72da-4936-87a5-0b476bb48d6f","ResponseTimeUtc":"2026-04-14T18:45:25.9857811Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 6702dc55-72da-4936-87a5-0b476bb48d6f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149888","BELatencyInMs":"0.709"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 18:46:04 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 464ae97a-fd9a-44af-88e3-d82585cede18; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 464ae97a-fd9a-44af-88e3-d82585cede18, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 464ae97a-fd9a-44af-88e3-d82585cede18; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 464ae97a-fd9a-44af-88e3-d82585cede18, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:46:03.986Z","duration in milliseconds":50.662800000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:46:03.8927606Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92769,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0456},{"name":"Get PkValue From Stream","duration in milliseconds":7.2107,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0013000000000000002},{"name":"Read Collection","duration in milliseconds":7.0550000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:46:03.9866374Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"4a65aede-7e29-4e80-955f-db409cdc3458","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":43.224000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":43.190000000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:45:10.1458115Z","cpu":28.627,"memory":5123440.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1341,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:45:20.1471954Z","cpu":10.391,"memory":5093228.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0616,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:45:30.1580174Z","cpu":4.375,"memory":5090264.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1164,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:45:40.1619697Z","cpu":1.794,"memory":5089544.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1398,"availableThreads":8188,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:45:50.1783564Z","cpu":5.772,"memory":5078908.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0827,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:46:00.1864299Z","cpu":10.078,"memory":5122468.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0793,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":43.182500000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":43.1786,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":43.1525,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":43.148700000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":42.3005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:46:04.0021364Z","EndTimeUTC":"2026-04-14T18:46:04.0021364Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:46:04.0333874Z","DurationInMs":31.251,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"464ae97a-fd9a-44af-88e3-d82585cede18","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149888,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149888,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149888,"SessionToken":"-1#26149888","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.595","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:46:04 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:46:04.0021364Z", "durationInMs": 0.0098},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:46:04.0021462Z", "durationInMs": 25.6314},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:46:04.0277776Z", "durationInMs": 0.2053},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:46:04.0279829Z", "durationInMs": 1.7928},{"event": "Received", "startTimeUtc": "2026-04-14T18:46:04.0297757Z", "durationInMs": 0.1491},{"event": "Completed", "startTimeUtc": "2026-04-14T18:46:04.0299248Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:46:04.0333874Z","lastSend":"2026-04-14T18:46:04.0333874Z","lastReceive":"2026-04-14T18:46:04.0333874Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:46:03.9866374Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"1edaec92-68bd-4647-81c7-787e6f32b7a3","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:46:03.9866374Z","DurationInMs":15.499,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"f93e1774-e242-4d26-a7d6-ff7ec89d05c6","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:46:04.0021364Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"464ae97a-fd9a-44af-88e3-d82585cede18","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1406268,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"464ae97a-fd9a-44af-88e3-d82585cede18","ResponseTimeUtc":"2026-04-14T18:46:04.0333874Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 464ae97a-fd9a-44af-88e3-d82585cede18, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149888","BELatencyInMs":"0.595"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 18:49:45 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 46205ea7-eaca-48f9-bbc2-e1ae6ea048d9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 46205ea7-eaca-48f9-bbc2-e1ae6ea048d9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 46205ea7-eaca-48f9-bbc2-e1ae6ea048d9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 46205ea7-eaca-48f9-bbc2-e1ae6ea048d9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:49:45.640Z","duration in milliseconds":47.4108,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:49:45.5158363Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92846,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0409},{"name":"Get PkValue From Stream","duration in milliseconds":6.8869000000000007,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.7482000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:49:45.6409480Z","DurationInMs":15.5249,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"637c2ebf-9ab3-4389-9fe4-39b3732da41d","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":40.3106,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":40.284400000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:48:50.3613250Z","cpu":1.326,"memory":5287876.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0937,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:49:00.3719245Z","cpu":3.666,"memory":5285300.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1143,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:49:10.3768124Z","cpu":2.031,"memory":5289124.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":1.5354,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:49:20.3782859Z","cpu":3.750,"memory":5291968.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1144,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:49:30.3852209Z","cpu":2.262,"memory":5291912.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1342,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:49:40.3964676Z","cpu":2.500,"memory":5283528.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0907,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":40.278,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":40.2725,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":40.255900000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":40.2535,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":39.089,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:49:45.6564729Z","EndTimeUTC":"2026-04-14T18:49:45.6721412Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:49:45.6877188Z","DurationInMs":15.5776,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"46205ea7-eaca-48f9-bbc2-e1ae6ea048d9","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149890,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149890,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149890,"SessionToken":"-1#26149890","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.699","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:49:45 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:49:45.6721412Z", "durationInMs": 0.0059},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:49:45.6721471Z", "durationInMs": 19.6203},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:49:45.6917674Z", "durationInMs": 0.1983},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:49:45.6919657Z", "durationInMs": 2.229},{"event": "Received", "startTimeUtc": "2026-04-14T18:49:45.6941947Z", "durationInMs": 0.2111},{"event": "Completed", "startTimeUtc": "2026-04-14T18:49:45.6944058Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:49:45.6877188Z","lastSend":"2026-04-14T18:49:45.6877188Z","lastReceive":"2026-04-14T18:49:45.6877188Z"},"requestSizeInBytes":1640,"requestBodySizeInBytes":1190,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:49:45.6564729Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"fb239d70-fc4f-42a4-a0ca-00ef1576bd03","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:49:45.6564729Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"c031ad6e-8ec5-406b-8fc2-3f73d922b338","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:49:45.6564729Z","DurationInMs":15.6683,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"46205ea7-eaca-48f9-bbc2-e1ae6ea048d9","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718825,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"46205ea7-eaca-48f9-bbc2-e1ae6ea048d9","ResponseTimeUtc":"2026-04-14T18:49:45.6877188Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 46205ea7-eaca-48f9-bbc2-e1ae6ea048d9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149890","BELatencyInMs":"0.699"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 18:50:05 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 066055cd-bc56-42db-baf0-b7ef098f7160; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 066055cd-bc56-42db-baf0-b7ef098f7160, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 066055cd-bc56-42db-baf0-b7ef098f7160; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 066055cd-bc56-42db-baf0-b7ef098f7160, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:50:05.362Z","duration in milliseconds":37.336200000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:50:04.9410077Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92853,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0526},{"name":"Get PkValue From Stream","duration in milliseconds":0.1598,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0023}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.9515,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.9152,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:49:10.3768124Z","cpu":2.031,"memory":5289124.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":1.5354,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:49:20.3782859Z","cpu":3.750,"memory":5291968.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1144,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:49:30.3852209Z","cpu":2.262,"memory":5291912.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1342,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:49:40.3964676Z","cpu":2.500,"memory":5283528.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0907,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:49:50.4107163Z","cpu":4.134,"memory":5283424.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1219,"availableThreads":8188,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:50:00.4226316Z","cpu":2.496,"memory":5282908.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1109,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.908100000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.9033,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.8838,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.8811,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.0099,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:50:05.4097041Z","DurationInMs":46.8742,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"066055cd-bc56-42db-baf0-b7ef098f7160","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149891,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149891,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149891,"SessionToken":"-1#26149891","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.614","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:50:05 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:50:05.3628299Z", "durationInMs": 0.0133},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:50:05.3628432Z", "durationInMs": 32.904},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:50:05.3957472Z", "durationInMs": 0.2182},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:50:05.3959654Z", "durationInMs": 1.9531},{"event": "Received", "startTimeUtc": "2026-04-14T18:50:05.3979185Z", "durationInMs": 0.1441},{"event": "Completed", "startTimeUtc": "2026-04-14T18:50:05.3980626Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:50:05.3940826Z","lastSend":"2026-04-14T18:50:05.3940826Z","lastReceive":"2026-04-14T18:50:05.3940826Z"},"requestSizeInBytes":1656,"requestBodySizeInBytes":1208,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.46869639999999996,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"066055cd-bc56-42db-baf0-b7ef098f7160","ResponseTimeUtc":"2026-04-14T18:50:05.4097041Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 066055cd-bc56-42db-baf0-b7ef098f7160, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149891","BELatencyInMs":"0.614"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 18:50:22 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 301b2b03-ee58-418a-b330-cc1e1862bb78; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 301b2b03-ee58-418a-b330-cc1e1862bb78, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 301b2b03-ee58-418a-b330-cc1e1862bb78; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 301b2b03-ee58-418a-b330-cc1e1862bb78, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:50:22.241Z","duration in milliseconds":22.8533,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:50:22.0385645Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92864,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.052000000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":0.1635,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":22.4543,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":22.418300000000002,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:49:30.3852209Z","cpu":2.262,"memory":5291912.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1342,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:49:40.3964676Z","cpu":2.500,"memory":5283528.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0907,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:49:50.4107163Z","cpu":4.134,"memory":5283424.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1219,"availableThreads":8188,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:50:00.4226316Z","cpu":2.496,"memory":5282908.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1109,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:50:10.4306965Z","cpu":5.000,"memory":5214292.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1334,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:50:20.4448745Z","cpu":25.663,"memory":5127256.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1221,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":22.408900000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":22.403100000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":22.374200000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":22.3713,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":21.0989,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:50:22.2729429Z","DurationInMs":31.125300000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"301b2b03-ee58-418a-b330-cc1e1862bb78","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149893,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149893,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149893,"SessionToken":"-1#26149893","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.607","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:50:22 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:50:22.2418176Z", "durationInMs": 0.0171},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:50:22.2418347Z", "durationInMs": 17.9434},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:50:22.2597781Z", "durationInMs": 0.1948},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:50:22.2599729Z", "durationInMs": 2.0372},{"event": "Received", "startTimeUtc": "2026-04-14T18:50:22.2620101Z", "durationInMs": 0.1256},{"event": "Completed", "startTimeUtc": "2026-04-14T18:50:22.2621357Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:50:22.2573177Z","lastSend":"2026-04-14T18:50:22.2573177Z","lastReceive":"2026-04-14T18:50:22.2573177Z"},"requestSizeInBytes":1664,"requestBodySizeInBytes":1210,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.2343784,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"301b2b03-ee58-418a-b330-cc1e1862bb78","ResponseTimeUtc":"2026-04-14T18:50:22.2729429Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 301b2b03-ee58-418a-b330-cc1e1862bb78, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149893","BELatencyInMs":"0.607"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 18:50:25 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 4df0aabe-a7bf-4e56-ab82-f86a3bd8b889; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 4df0aabe-a7bf-4e56-ab82-f86a3bd8b889, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 4df0aabe-a7bf-4e56-ab82-f86a3bd8b889; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 4df0aabe-a7bf-4e56-ab82-f86a3bd8b889, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:50:25.944Z","duration in milliseconds":40.175000000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:50:25.8198977Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92865,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0383},{"name":"Get PkValue From Stream","duration in milliseconds":6.4551000000000007,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.001},{"name":"Read Collection","duration in milliseconds":6.3051,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:50:25.9448998Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"6545acdc-49f2-4551-9247-3fe9b8876e2a","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":33.517,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":33.4816,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:49:30.3852209Z","cpu":2.262,"memory":5291912.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1342,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:49:40.3964676Z","cpu":2.500,"memory":5283528.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0907,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:49:50.4107163Z","cpu":4.134,"memory":5283424.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1219,"availableThreads":8188,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:50:00.4226316Z","cpu":2.496,"memory":5282908.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1109,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:50:10.4306965Z","cpu":5.000,"memory":5214292.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1334,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:50:20.4448745Z","cpu":25.663,"memory":5127256.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1221,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":33.4752,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":33.470800000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":33.4523,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":33.4501,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":32.3327,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:50:25.9605175Z","EndTimeUTC":"2026-04-14T18:50:25.9605175Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:50:25.9761505Z","DurationInMs":15.633000000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"4df0aabe-a7bf-4e56-ab82-f86a3bd8b889","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149893,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149893,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149893,"SessionToken":"-1#26149893","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.566","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:50:25 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:50:25.9605175Z", "durationInMs": 0.0086},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:50:25.9605261Z", "durationInMs": 14.7977},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:50:25.9753238Z", "durationInMs": 0.1913},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:50:25.9755151Z", "durationInMs": 2.0741},{"event": "Received", "startTimeUtc": "2026-04-14T18:50:25.9775892Z", "durationInMs": 0.122},{"event": "Completed", "startTimeUtc": "2026-04-14T18:50:25.9777112Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:50:25.9761505Z","lastSend":"2026-04-14T18:50:25.9761505Z","lastReceive":"2026-04-14T18:50:25.9761505Z"},"requestSizeInBytes":1638,"requestBodySizeInBytes":1190,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:50:25.9448998Z","DurationInMs":15.617700000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"fca2820b-4ffb-4398-b9ff-a7ba1365b370","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:50:25.9605175Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"8477337b-92ce-4a70-a601-84b54d6fa7b8","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:50:25.9605175Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"4df0aabe-a7bf-4e56-ab82-f86a3bd8b889","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1562528,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"4df0aabe-a7bf-4e56-ab82-f86a3bd8b889","ResponseTimeUtc":"2026-04-14T18:50:25.9917720Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 4df0aabe-a7bf-4e56-ab82-f86a3bd8b889, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149893","BELatencyInMs":"0.566"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 18:51:27 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 7714f011-1413-43e6-a140-89d42601646f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 7714f011-1413-43e6-a140-89d42601646f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 7714f011-1413-43e6-a140-89d42601646f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 7714f011-1413-43e6-a140-89d42601646f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:51:27.712Z","duration in milliseconds":43.452000000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:51:27.5874570Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92876,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0439},{"name":"Get PkValue From Stream","duration in milliseconds":6.3041,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0034000000000000002},{"name":"Read Collection","duration in milliseconds":6.1632000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:51:27.7124585Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"5e2cca37-2fbf-48c9-ae25-704145b1c6c0","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.933,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.9039,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:50:30.4523435Z","cpu":4.134,"memory":5142220.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0953,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:50:40.4559132Z","cpu":1.484,"memory":5142244.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0952,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:50:50.4630946Z","cpu":1.406,"memory":5145756.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1152,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:51:00.4662343Z","cpu":2.574,"memory":5146152.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0501,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:51:10.4678861Z","cpu":2.734,"memory":5145784.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0865,"availableThreads":8188,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:51:20.4719580Z","cpu":2.109,"memory":5146524.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0882,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.897,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.8937,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.874,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.8714,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.8161,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:51:27.7280845Z","EndTimeUTC":"2026-04-14T18:51:27.7280845Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:51:27.7437090Z","DurationInMs":15.624500000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"7714f011-1413-43e6-a140-89d42601646f","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149893,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149893,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149893,"SessionToken":"-1#26149893","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"1.342","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:51:27 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:51:27.7280845Z", "durationInMs": 0.0072},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:51:27.7280917Z", "durationInMs": 19.4291},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:51:27.7475208Z", "durationInMs": 0.2047},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:51:27.7477255Z", "durationInMs": 3.1495},{"event": "Received", "startTimeUtc": "2026-04-14T18:51:27.7508750Z", "durationInMs": 0.136},{"event": "Completed", "startTimeUtc": "2026-04-14T18:51:27.7510110Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:51:27.7437090Z","lastSend":"2026-04-14T18:51:27.7437090Z","lastReceive":"2026-04-14T18:51:27.7437090Z"},"requestSizeInBytes":1646,"requestBodySizeInBytes":1192,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:51:27.7124585Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"b48f7994-4bf0-45fc-a1d0-49d8d405b269","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:51:27.7124585Z","DurationInMs":15.626000000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"2840b1b0-a96f-4f18-990b-4e584a19206e","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:51:27.7280845Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"7714f011-1413-43e6-a140-89d42601646f","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.156252,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"7714f011-1413-43e6-a140-89d42601646f","ResponseTimeUtc":"2026-04-14T18:51:27.7437090Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 7714f011-1413-43e6-a140-89d42601646f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149893","BELatencyInMs":"1.342"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0014}]}
2026-04-14 18:54:47 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 18c91800-bf17-4ae7-9e86-8ba0abb6c98d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 18c91800-bf17-4ae7-9e86-8ba0abb6c98d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 18c91800-bf17-4ae7-9e86-8ba0abb6c98d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 18c91800-bf17-4ae7-9e86-8ba0abb6c98d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:54:47.427Z","duration in milliseconds":41.0792,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:54:47.3020607Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92906,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0563},{"name":"Get PkValue From Stream","duration in milliseconds":6.4131,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014},{"name":"Read Collection","duration in milliseconds":6.2472,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:54:47.4277546Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"37a53327-a4b2-46ea-8393-f5d88bab50bb","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":34.4292,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":34.394600000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:53:50.6348379Z","cpu":0.702,"memory":5281608.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1069,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:54:00.6385508Z","cpu":1.328,"memory":5279556.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1373,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:54:10.6402424Z","cpu":4.758,"memory":5268276.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0992,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:54:20.6409655Z","cpu":0.391,"memory":5266828.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1144,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:54:30.6562468Z","cpu":3.588,"memory":5269920.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1012,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:54:40.6640624Z","cpu":2.578,"memory":5264872.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1007,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":34.3873,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":34.382600000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":34.363800000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":34.3615,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":33.4,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:54:47.4432519Z","EndTimeUTC":"2026-04-14T18:54:47.4432519Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:54:47.4744976Z","DurationInMs":31.245700000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"18c91800-bf17-4ae7-9e86-8ba0abb6c98d","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149899,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149899,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149899,"SessionToken":"-1#26149899","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.587","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:54:47 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:54:47.4432519Z", "durationInMs": 0.0133},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:54:47.4432652Z", "durationInMs": 16.7402},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:54:47.4600054Z", "durationInMs": 0.2251},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:54:47.4602305Z", "durationInMs": 1.6745},{"event": "Received", "startTimeUtc": "2026-04-14T18:54:47.4619050Z", "durationInMs": 0.1404},{"event": "Completed", "startTimeUtc": "2026-04-14T18:54:47.4620454Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:54:47.4588741Z","lastSend":"2026-04-14T18:54:47.4588741Z","lastReceive":"2026-04-14T18:54:47.4588741Z"},"requestSizeInBytes":1638,"requestBodySizeInBytes":1190,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:54:47.4277546Z","DurationInMs":15.497300000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a233d674-a091-441f-b368-fba1608ce7c1","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:54:47.4432519Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"b0abb02a-b591-4e0e-a126-01b4ce0664ee","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:54:47.4432519Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"18c91800-bf17-4ae7-9e86-8ba0abb6c98d","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1724369,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"18c91800-bf17-4ae7-9e86-8ba0abb6c98d","ResponseTimeUtc":"2026-04-14T18:54:47.4744976Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 18c91800-bf17-4ae7-9e86-8ba0abb6c98d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149899","BELatencyInMs":"0.587"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 18:58:19 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 8e1be9c3-26de-46ce-a394-fa96a5528c70; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 8e1be9c3-26de-46ce-a394-fa96a5528c70, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 8e1be9c3-26de-46ce-a394-fa96a5528c70; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 8e1be9c3-26de-46ce-a394-fa96a5528c70, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:58:19.036Z","duration in milliseconds":52.3403,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:58:18.1922959Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92927,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.038},{"name":"Get PkValue From Stream","duration in milliseconds":12.3465,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0042000000000000006},{"name":"Read Collection","duration in milliseconds":12.1921,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:58:19.0360487Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"d15d5edf-c1a8-48e9-ab0e-4ff617658ecc","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":39.7779,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":39.7417,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:57:20.8183553Z","cpu":2.496,"memory":5174976.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1037,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:57:30.8202806Z","cpu":3.281,"memory":5176020.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0909,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:57:40.8302290Z","cpu":0.781,"memory":5175760.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0976,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:57:50.8347796Z","cpu":1.092,"memory":5184644.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1184,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:58:00.8383256Z","cpu":2.266,"memory":5273600.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0832,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:58:10.8477379Z","cpu":1.794,"memory":5269348.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.074,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":39.7321,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":39.7261,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":39.703900000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":39.7016,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":38.7121,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T18:58:19.0516210Z","EndTimeUTC":"2026-04-14T18:58:19.0516210Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:58:19.0828708Z","DurationInMs":31.2498,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"8e1be9c3-26de-46ce-a394-fa96a5528c70","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149913,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149913,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149913,"SessionToken":"-1#26149913","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.629","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:58:19 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:58:19.0516210Z", "durationInMs": 0.0097},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:58:19.0516307Z", "durationInMs": 21.2563},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:58:19.0728870Z", "durationInMs": 0.2189},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:58:19.0731059Z", "durationInMs": 2.1578},{"event": "Received", "startTimeUtc": "2026-04-14T18:58:19.0752637Z", "durationInMs": 0.1464},{"event": "Completed", "startTimeUtc": "2026-04-14T18:58:19.0754101Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:58:19.0828708Z","lastSend":"2026-04-14T18:58:19.0828708Z","lastReceive":"2026-04-14T18:58:19.0828708Z"},"requestSizeInBytes":1628,"requestBodySizeInBytes":1174,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T18:58:19.0360487Z","DurationInMs":15.5723,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"2c65c1d3-5f6a-4f54-8c6a-094c3098cbe8","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T18:58:19.0516210Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"b1716e85-57f5-4937-b3c3-c9bb521d17c9","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T18:58:19.0516210Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"8e1be9c3-26de-46ce-a394-fa96a5528c70","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.8905749,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"8e1be9c3-26de-46ce-a394-fa96a5528c70","ResponseTimeUtc":"2026-04-14T18:58:19.0828708Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 8e1be9c3-26de-46ce-a394-fa96a5528c70, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149913","BELatencyInMs":"0.629"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001}]}
2026-04-14 18:58:36 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d23efa9f-a1b3-4cc8-b967-e9c99e9b6656; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d23efa9f-a1b3-4cc8-b967-e9c99e9b6656, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d23efa9f-a1b3-4cc8-b967-e9c99e9b6656; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d23efa9f-a1b3-4cc8-b967-e9c99e9b6656, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:58:36.809Z","duration in milliseconds":36.3177,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:58:36.6381321Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92938,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.051500000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":0.1632,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.9328,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.8994,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:57:40.8302290Z","cpu":0.781,"memory":5175760.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0976,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:57:50.8347796Z","cpu":1.092,"memory":5184644.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1184,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:58:00.8383256Z","cpu":2.266,"memory":5273600.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0832,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:58:10.8477379Z","cpu":1.794,"memory":5269348.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.074,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:58:20.8513843Z","cpu":3.828,"memory":5264592.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1789,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:58:30.8649973Z","cpu":2.028,"memory":5273512.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0621,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.891200000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.886500000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.8701,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.8678,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.008,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:58:36.8568159Z","DurationInMs":46.877100000000006,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"d23efa9f-a1b3-4cc8-b967-e9c99e9b6656","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149913,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149913,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149913,"SessionToken":"-1#26149913","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.521","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:58:36 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:58:36.8099388Z", "durationInMs": 0.0134},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:58:36.8099522Z", "durationInMs": 32.0661},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:58:36.8420183Z", "durationInMs": 0.2141},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:58:36.8422324Z", "durationInMs": 1.7702},{"event": "Received", "startTimeUtc": "2026-04-14T18:58:36.8440026Z", "durationInMs": 0.1448},{"event": "Completed", "startTimeUtc": "2026-04-14T18:58:36.8441474Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:58:36.8412416Z","lastSend":"2026-04-14T18:58:36.8412416Z","lastReceive":"2026-04-14T18:58:36.8412416Z"},"requestSizeInBytes":1662,"requestBodySizeInBytes":1210,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.21868379999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"d23efa9f-a1b3-4cc8-b967-e9c99e9b6656","ResponseTimeUtc":"2026-04-14T18:58:36.8568159Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: d23efa9f-a1b3-4cc8-b967-e9c99e9b6656, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149913","BELatencyInMs":"0.521"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 18:59:59 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d8b93dc2-7de6-4063-9a4d-1615e725e46a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d8b93dc2-7de6-4063-9a4d-1615e725e46a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d8b93dc2-7de6-4063-9a4d-1615e725e46a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d8b93dc2-7de6-4063-9a4d-1615e725e46a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T18:59:59.675Z","duration in milliseconds":24.2802,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T18:59:59.3785194Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92963,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0557},{"name":"Get PkValue From Stream","duration in milliseconds":0.16770000000000002,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0022}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":23.8744,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":23.8426,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:59:00.8854995Z","cpu":0.469,"memory":5263968.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0917,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:59:10.8973755Z","cpu":3.900,"memory":5265432.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1023,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:59:20.9017130Z","cpu":3.594,"memory":5266588.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1034,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:59:30.9047136Z","cpu":1.484,"memory":5263736.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0848,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:59:40.9155383Z","cpu":0.624,"memory":5259848.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1126,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:59:50.9167418Z","cpu":0.703,"memory":5263140.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0799,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":23.835,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":23.8309,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":23.8123,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":23.8096,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":22.8402,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T18:59:59.7066471Z","DurationInMs":31.244,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"d8b93dc2-7de6-4063-9a4d-1615e725e46a","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149913,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149913,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149913,"SessionToken":"-1#26149913","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.688","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 6:59:59 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T18:59:59.6754031Z", "durationInMs": 0.0149},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T18:59:59.6754180Z", "durationInMs": 19.7244},{"event": "Pipelined", "startTimeUtc": "2026-04-14T18:59:59.6951424Z", "durationInMs": 0.1947},{"event": "Transit Time", "startTimeUtc": "2026-04-14T18:59:59.6953371Z", "durationInMs": 1.9166},{"event": "Received", "startTimeUtc": "2026-04-14T18:59:59.6972537Z", "durationInMs": 0.1365},{"event": "Completed", "startTimeUtc": "2026-04-14T18:59:59.6973902Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T18:59:59.6910903Z","lastSend":"2026-04-14T18:59:59.6910903Z","lastReceive":"2026-04-14T18:59:59.6910903Z"},"requestSizeInBytes":1657,"requestBodySizeInBytes":1207,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.32812769999999997,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"d8b93dc2-7de6-4063-9a4d-1615e725e46a","ResponseTimeUtc":"2026-04-14T18:59:59.7066471Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: d8b93dc2-7de6-4063-9a4d-1615e725e46a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149913","BELatencyInMs":"0.688"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0023}]}
2026-04-14 19:00:34 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 08e54a8b-6063-4b62-8665-eb9c72836d7e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 08e54a8b-6063-4b62-8665-eb9c72836d7e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 08e54a8b-6063-4b62-8665-eb9c72836d7e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 08e54a8b-6063-4b62-8665-eb9c72836d7e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T19:00:34.894Z","duration in milliseconds":54.2331,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T19:00:34.6288648Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":92983,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.033800000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":7.3971,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.00090000000000000008},{"name":"Read Collection","duration in milliseconds":7.1928,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:00:34.8945498Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"29a470e1-deb9-406f-a979-376fb30952b0","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":46.6299,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":46.5993,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T18:59:40.9155383Z","cpu":0.624,"memory":5259848.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1126,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T18:59:50.9167418Z","cpu":0.703,"memory":5263140.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0799,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:00:00.9303539Z","cpu":3.198,"memory":5355680.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1439,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:00:10.9365494Z","cpu":11.250,"memory":5272532.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1193,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:00:20.9413587Z","cpu":4.453,"memory":5271968.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0774,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:00:30.9427754Z","cpu":5.234,"memory":5287480.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1156,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":46.5927,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":46.5893,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":46.5689,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":46.5664,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":45.6122,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T19:00:34.9101192Z","EndTimeUTC":"2026-04-14T19:00:34.9101192Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T19:00:34.9413697Z","DurationInMs":31.250500000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"08e54a8b-6063-4b62-8665-eb9c72836d7e","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149913,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149913,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149913,"SessionToken":"-1#26149913","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.693","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 7:00:34 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T19:00:34.9101192Z", "durationInMs": 0.0074},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T19:00:34.9101266Z", "durationInMs": 29.8698},{"event": "Pipelined", "startTimeUtc": "2026-04-14T19:00:34.9399964Z", "durationInMs": 0.2047},{"event": "Transit Time", "startTimeUtc": "2026-04-14T19:00:34.9402011Z", "durationInMs": 1.8779},{"event": "Received", "startTimeUtc": "2026-04-14T19:00:34.9420790Z", "durationInMs": 0.1451},{"event": "Completed", "startTimeUtc": "2026-04-14T19:00:34.9422241Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T19:00:34.9413697Z","lastSend":"2026-04-14T19:00:34.9413697Z","lastReceive":"2026-04-14T19:00:34.9413697Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1186,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:00:34.8945498Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"6026a11a-eac4-45bf-ad29-d5083e38f18e","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T19:00:34.8945498Z","DurationInMs":15.5694,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"067b429a-329f-4d06-b547-d5af91cca2d4","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T19:00:34.9101192Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"08e54a8b-6063-4b62-8665-eb9c72836d7e","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.3125049,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"08e54a8b-6063-4b62-8665-eb9c72836d7e","ResponseTimeUtc":"2026-04-14T19:00:34.9413697Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 08e54a8b-6063-4b62-8665-eb9c72836d7e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149913","BELatencyInMs":"0.693"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0029000000000000002}]}
2026-04-14 19:06:03 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5d7b8633-ce3f-4281-99be-ce30d9ac7764; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5d7b8633-ce3f-4281-99be-ce30d9ac7764, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5d7b8633-ce3f-4281-99be-ce30d9ac7764; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5d7b8633-ce3f-4281-99be-ce30d9ac7764, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T19:06:03.143Z","duration in milliseconds":18.6274,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T19:06:02.9558553Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93060,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0514},{"name":"Get PkValue From Stream","duration in milliseconds":0.166,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0049}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":18.2348,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":18.2026,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T19:05:11.2170563Z","cpu":1.560,"memory":5302288.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1052,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:05:21.2287674Z","cpu":3.984,"memory":5289472.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1418,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:05:31.2318561Z","cpu":1.014,"memory":5288036.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1081,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:05:41.2345541Z","cpu":1.875,"memory":5287440.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0548,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:05:51.2399682Z","cpu":1.563,"memory":5286268.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1224,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:06:01.2420401Z","cpu":3.047,"memory":5243892.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1023,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":18.1952,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":18.1904,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":18.1736,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":18.171300000000002,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":17.201,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T19:06:03.1589842Z","DurationInMs":15.6239,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"5d7b8633-ce3f-4281-99be-ce30d9ac7764","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26149944,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26149944,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26149944,"SessionToken":"-1#26149944","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.693","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 7:06:03 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T19:06:03.1433603Z", "durationInMs": 0.0172},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T19:06:03.1433775Z", "durationInMs": 14.2184},{"event": "Pipelined", "startTimeUtc": "2026-04-14T19:06:03.1575959Z", "durationInMs": 0.1979},{"event": "Transit Time", "startTimeUtc": "2026-04-14T19:06:03.1577938Z", "durationInMs": 1.872},{"event": "Received", "startTimeUtc": "2026-04-14T19:06:03.1596658Z", "durationInMs": 0.1426},{"event": "Completed", "startTimeUtc": "2026-04-14T19:06:03.1598084Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T19:06:03.1589842Z","lastSend":"2026-04-14T19:06:03.1589842Z","lastReceive":"2026-04-14T19:06:03.1589842Z"},"requestSizeInBytes":1639,"requestBodySizeInBytes":1187,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.2031289,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"5d7b8633-ce3f-4281-99be-ce30d9ac7764","ResponseTimeUtc":"2026-04-14T19:06:03.1589842Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 5d7b8633-ce3f-4281-99be-ce30d9ac7764, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26149944","BELatencyInMs":"0.693"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 19:18:47 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: a18531c2-a6d1-4365-91e9-b5495d02a8a9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: a18531c2-a6d1-4365-91e9-b5495d02a8a9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: a18531c2-a6d1-4365-91e9-b5495d02a8a9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: a18531c2-a6d1-4365-91e9-b5495d02a8a9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T19:18:47.046Z","duration in milliseconds":42.4406,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T19:18:46.9523535Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93161,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.046900000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":7.394,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014},{"name":"Read Collection","duration in milliseconds":7.2273000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:18:47.0461078Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"db42a157-72ff-424e-968a-dd3af981b050","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":34.7352,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":34.6946,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T19:17:51.8452201Z","cpu":1.641,"memory":5204456.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0958,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:18:01.8611785Z","cpu":1.326,"memory":5257776.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0584,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:18:11.8710409Z","cpu":0.780,"memory":5251544.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1117,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:18:21.8793698Z","cpu":2.344,"memory":5272532.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1162,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:18:31.8898713Z","cpu":5.772,"memory":5265184.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1035,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:18:41.9044444Z","cpu":2.808,"memory":5263980.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0859,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":34.681400000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":34.6668,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":34.596000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":34.592800000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":33.61,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T19:18:47.0617916Z","EndTimeUTC":"2026-04-14T19:18:47.0617916Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T19:18:47.0773556Z","DurationInMs":15.564,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"a18531c2-a6d1-4365-91e9-b5495d02a8a9","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150043,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150043,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150043,"SessionToken":"-1#26150043","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.754","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 7:18:47 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T19:18:47.0617916Z", "durationInMs": 0.0083},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T19:18:47.0617999Z", "durationInMs": 15.3111},{"event": "Pipelined", "startTimeUtc": "2026-04-14T19:18:47.0771110Z", "durationInMs": 0.2027},{"event": "Transit Time", "startTimeUtc": "2026-04-14T19:18:47.0773137Z", "durationInMs": 2.0451},{"event": "Received", "startTimeUtc": "2026-04-14T19:18:47.0793588Z", "durationInMs": 0.1375},{"event": "Completed", "startTimeUtc": "2026-04-14T19:18:47.0794963Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T19:18:47.0773556Z","lastSend":"2026-04-14T19:18:47.0773556Z","lastReceive":"2026-04-14T19:18:47.0773556Z"},"requestSizeInBytes":1639,"requestBodySizeInBytes":1187,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:18:47.0461078Z","DurationInMs":15.683800000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"c21fe158-e867-451d-bd3f-62da44e516de","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T19:18:47.0617916Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"e219565c-339b-400a-9e9b-aa8104aa24ce","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T19:18:47.0617916Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"a18531c2-a6d1-4365-91e9-b5495d02a8a9","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1250021,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"a18531c2-a6d1-4365-91e9-b5495d02a8a9","ResponseTimeUtc":"2026-04-14T19:18:47.0773556Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: a18531c2-a6d1-4365-91e9-b5495d02a8a9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150043","BELatencyInMs":"0.754"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0036000000000000003}]}
2026-04-14 19:25:53 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c539f693-165a-4131-88b5-53856445c2b7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c539f693-165a-4131-88b5-53856445c2b7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c539f693-165a-4131-88b5-53856445c2b7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c539f693-165a-4131-88b5-53856445c2b7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T19:25:53.573Z","duration in milliseconds":31.287200000000002,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T19:25:53.4333492Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93248,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0523},{"name":"Get PkValue From Stream","duration in milliseconds":0.1665,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":30.892100000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":30.8561,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T19:25:02.2502978Z","cpu":0.938,"memory":5304608.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0716,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:25:12.2569122Z","cpu":2.340,"memory":5287932.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0808,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:25:22.2720035Z","cpu":0.390,"memory":5286284.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0927,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:25:32.2731131Z","cpu":3.672,"memory":5275428.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1054,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:25:42.2870648Z","cpu":3.120,"memory":5277992.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0716,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:25:52.2928073Z","cpu":1.797,"memory":5275628.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0779,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":30.848100000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":30.8431,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":30.8259,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":30.823500000000003,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":29.8598,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T19:25:53.6053047Z","DurationInMs":31.3255,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c539f693-165a-4131-88b5-53856445c2b7","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150064,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150064,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150064,"SessionToken":"-1#26150064","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.895","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 7:25:53 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T19:25:53.5739792Z", "durationInMs": 0.0139},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T19:25:53.5739931Z", "durationInMs": 26.2114},{"event": "Pipelined", "startTimeUtc": "2026-04-14T19:25:53.6002045Z", "durationInMs": 0.2227},{"event": "Transit Time", "startTimeUtc": "2026-04-14T19:25:53.6004272Z", "durationInMs": 2.4972},{"event": "Received", "startTimeUtc": "2026-04-14T19:25:53.6029244Z", "durationInMs": 0.1469},{"event": "Completed", "startTimeUtc": "2026-04-14T19:25:53.6030713Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T19:25:53.5896026Z","lastSend":"2026-04-14T19:25:53.5896026Z","lastReceive":"2026-04-14T19:25:53.6053047Z"},"requestSizeInBytes":1656,"requestBodySizeInBytes":1208,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.17195549999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c539f693-165a-4131-88b5-53856445c2b7","ResponseTimeUtc":"2026-04-14T19:25:53.6053047Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c539f693-165a-4131-88b5-53856445c2b7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150064","BELatencyInMs":"0.895"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 19:26:40 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 3fb37376-d913-46e6-aac5-c5328d3d7699; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 3fb37376-d913-46e6-aac5-c5328d3d7699, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 3fb37376-d913-46e6-aac5-c5328d3d7699; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 3fb37376-d913-46e6-aac5-c5328d3d7699, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T19:26:40.052Z","duration in milliseconds":45.7853,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T19:26:39.7243620Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93257,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0483},{"name":"Get PkValue From Stream","duration in milliseconds":7.1299,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0008},{"name":"Read Collection","duration in milliseconds":6.8809000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:26:40.0524900Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"eb40859b-0a53-48b8-a809-ccbaa1806d38","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":38.4333,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":38.392900000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T19:25:42.2870648Z","cpu":3.120,"memory":5277992.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0716,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:25:52.2928073Z","cpu":1.797,"memory":5275628.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0779,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:26:02.2961722Z","cpu":5.234,"memory":5224096.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0589,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:26:12.3034641Z","cpu":9.282,"memory":5162368.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1088,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:26:22.3148140Z","cpu":23.047,"memory":5150636.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1021,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":6},{"dateUtc":"2026-04-14T19:26:32.3219446Z","cpu":0.702,"memory":5144092.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0445,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":38.385600000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":38.3817,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":38.3562,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":38.3534,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":37.150200000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T19:26:40.0681295Z","EndTimeUTC":"2026-04-14T19:26:40.0681295Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T19:26:40.0837382Z","DurationInMs":15.6087,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"3fb37376-d913-46e6-aac5-c5328d3d7699","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150071,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150071,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150071,"SessionToken":"-1#26150071","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.583","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 7:26:40 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T19:26:40.0681295Z", "durationInMs": 0.0101},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T19:26:40.0681396Z", "durationInMs": 18.6049},{"event": "Pipelined", "startTimeUtc": "2026-04-14T19:26:40.0867445Z", "durationInMs": 0.2622},{"event": "Transit Time", "startTimeUtc": "2026-04-14T19:26:40.0870067Z", "durationInMs": 1.9391},{"event": "Received", "startTimeUtc": "2026-04-14T19:26:40.0889458Z", "durationInMs": 0.1508},{"event": "Completed", "startTimeUtc": "2026-04-14T19:26:40.0890966Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T19:26:40.0837382Z","lastSend":"2026-04-14T19:26:40.0837382Z","lastReceive":"2026-04-14T19:26:40.0837382Z"},"requestSizeInBytes":1637,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:26:40.0524900Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"596d625b-6813-43fb-9101-cac84352830a","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T19:26:40.0524900Z","DurationInMs":15.6395,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"bd5a3d4e-d79f-4a7f-afbe-da968543a928","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T19:26:40.0681295Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"3fb37376-d913-46e6-aac5-c5328d3d7699","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.3593762,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"3fb37376-d913-46e6-aac5-c5328d3d7699","ResponseTimeUtc":"2026-04-14T19:26:40.0837382Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 3fb37376-d913-46e6-aac5-c5328d3d7699, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150071","BELatencyInMs":"0.583"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0014}]}
2026-04-14 19:27:44 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 722254d3-95ff-4214-90e8-ba1afe20367d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 722254d3-95ff-4214-90e8-ba1afe20367d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 722254d3-95ff-4214-90e8-ba1afe20367d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 722254d3-95ff-4214-90e8-ba1afe20367d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T19:27:44.376Z","duration in milliseconds":44.6334,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T19:27:44.2360242Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93273,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0454},{"name":"Get PkValue From Stream","duration in milliseconds":6.8542000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.7116000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:27:44.3767198Z","DurationInMs":15.556700000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"dbcc5775-2357-4d7c-89a5-7074003bdd82","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.5601,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.5324,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T19:26:52.3522608Z","cpu":16.849,"memory":5214576.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0632,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:27:02.3550335Z","cpu":3.281,"memory":5211108.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0636,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:27:12.3557784Z","cpu":2.109,"memory":5201144.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1329,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:27:22.3714918Z","cpu":2.496,"memory":5198980.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1137,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":3},{"dateUtc":"2026-04-14T19:27:32.3749204Z","cpu":2.891,"memory":5197220.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0487,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:27:42.3901723Z","cpu":0.702,"memory":5200100.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0992,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.5251,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.522200000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.5034,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.501200000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.545100000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T19:27:44.4079703Z","EndTimeUTC":"2026-04-14T19:27:44.4079703Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T19:27:44.4235268Z","DurationInMs":15.556500000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"722254d3-95ff-4214-90e8-ba1afe20367d","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150078,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150078,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150078,"SessionToken":"-1#26150078","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.575","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 7:27:44 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T19:27:44.4079703Z", "durationInMs": 0.0086},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T19:27:44.4079789Z", "durationInMs": 18.8601},{"event": "Pipelined", "startTimeUtc": "2026-04-14T19:27:44.4268390Z", "durationInMs": 0.2071},{"event": "Transit Time", "startTimeUtc": "2026-04-14T19:27:44.4270461Z", "durationInMs": 2.1821},{"event": "Received", "startTimeUtc": "2026-04-14T19:27:44.4292282Z", "durationInMs": 0.1428},{"event": "Completed", "startTimeUtc": "2026-04-14T19:27:44.4293710Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T19:27:44.4235268Z","lastSend":"2026-04-14T19:27:44.4235268Z","lastReceive":"2026-04-14T19:27:44.4235268Z"},"requestSizeInBytes":1641,"requestBodySizeInBytes":1187,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:27:44.3922765Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"1bef4fbe-3baa-4205-a5b3-06329f23e902","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T19:27:44.3922765Z","DurationInMs":15.693800000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"f666fd3c-c091-47dc-b1b1-291a33605643","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T19:27:44.4079703Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"722254d3-95ff-4214-90e8-ba1afe20367d","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1875026,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"722254d3-95ff-4214-90e8-ba1afe20367d","ResponseTimeUtc":"2026-04-14T19:27:44.4235268Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 722254d3-95ff-4214-90e8-ba1afe20367d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150078","BELatencyInMs":"0.575"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 19:38:40 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: cb8ef8c1-70cb-45a1-a3ee-67df0cfa8755; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: cb8ef8c1-70cb-45a1-a3ee-67df0cfa8755, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: cb8ef8c1-70cb-45a1-a3ee-67df0cfa8755; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: cb8ef8c1-70cb-45a1-a3ee-67df0cfa8755, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T19:38:40.192Z","duration in milliseconds":41.6905,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T19:38:39.4272041Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93374,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0511},{"name":"Get PkValue From Stream","duration in milliseconds":6.6447,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014},{"name":"Read Collection","duration in milliseconds":6.4529000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:38:40.1929048Z","DurationInMs":15.6216,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"8cee79a3-107b-4144-b6c7-376985752a06","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":34.808800000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":34.7637,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T19:37:42.9294444Z","cpu":2.031,"memory":5282028.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0886,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:37:52.9335304Z","cpu":1.953,"memory":5295248.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1008,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:38:02.9402684Z","cpu":5.382,"memory":5342972.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1246,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:38:12.9560123Z","cpu":2.106,"memory":5333028.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0902,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:38:22.9707521Z","cpu":3.047,"memory":5340056.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1255,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:38:32.9762822Z","cpu":1.404,"memory":5330764.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1782,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":34.7554,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":34.75,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":34.7241,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":34.7207,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":33.7369,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T19:38:40.2085264Z","EndTimeUTC":"2026-04-14T19:38:40.2085264Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T19:38:40.2397241Z","DurationInMs":15.6258,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"cb8ef8c1-70cb-45a1-a3ee-67df0cfa8755","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150113,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150113,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150113,"SessionToken":"-1#26150113","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.717","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 7:38:40 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T19:38:40.2240983Z", "durationInMs": 0.0099},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T19:38:40.2241082Z", "durationInMs": 15.7255},{"event": "Pipelined", "startTimeUtc": "2026-04-14T19:38:40.2398337Z", "durationInMs": 0.1852},{"event": "Transit Time", "startTimeUtc": "2026-04-14T19:38:40.2400189Z", "durationInMs": 2.2245},{"event": "Received", "startTimeUtc": "2026-04-14T19:38:40.2422434Z", "durationInMs": 0.1371},{"event": "Completed", "startTimeUtc": "2026-04-14T19:38:40.2423805Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T19:38:40.2240983Z","lastSend":"2026-04-14T19:38:40.2240983Z","lastReceive":"2026-04-14T19:38:40.2240983Z"},"requestSizeInBytes":1627,"requestBodySizeInBytes":1175,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:38:40.2085264Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"1188cde0-6667-434c-a7bc-4cdbc8a611e6","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T19:38:40.2085264Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"7d895dea-a5d1-494c-91a9-b3dec37a184c","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T19:38:40.2085264Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"cb8ef8c1-70cb-45a1-a3ee-67df0cfa8755","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.81251999999999991,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"cb8ef8c1-70cb-45a1-a3ee-67df0cfa8755","ResponseTimeUtc":"2026-04-14T19:38:40.2397241Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: cb8ef8c1-70cb-45a1-a3ee-67df0cfa8755, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150113","BELatencyInMs":"0.717"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.003}]}
2026-04-14 19:41:50 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: ac2a84f6-cd36-4601-999d-20289baadeb7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: ac2a84f6-cd36-4601-999d-20289baadeb7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: ac2a84f6-cd36-4601-999d-20289baadeb7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: ac2a84f6-cd36-4601-999d-20289baadeb7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T19:41:50.272Z","duration in milliseconds":45.1368,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T19:41:49.9752005Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93413,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0541},{"name":"Get PkValue From Stream","duration in milliseconds":7.5288,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0015},{"name":"Read Collection","duration in milliseconds":7.2939000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:41:50.2721262Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"ad8fc43d-99bc-4fad-a304-9bf13713997d","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.3592,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.3068,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T19:40:53.0967853Z","cpu":3.438,"memory":5264932.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0781,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:41:03.1008191Z","cpu":4.212,"memory":5260956.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0539,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:41:13.1133059Z","cpu":4.531,"memory":5259872.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1205,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:41:23.1138635Z","cpu":3.672,"memory":5263904.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0897,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:41:33.1187266Z","cpu":4.134,"memory":5212528.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1227,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:41:43.1209613Z","cpu":1.016,"memory":5209436.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1637,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.2948,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.2902,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.263200000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.2601,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.2997,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T19:41:50.2877169Z","EndTimeUTC":"2026-04-14T19:41:50.3033303Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T19:41:50.3189595Z","DurationInMs":15.6292,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"ac2a84f6-cd36-4601-999d-20289baadeb7","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150125,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150125,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150125,"SessionToken":"-1#26150125","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.607","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 7:41:50 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T19:41:50.3033303Z", "durationInMs": 0.0082},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T19:41:50.3033385Z", "durationInMs": 17.3524},{"event": "Pipelined", "startTimeUtc": "2026-04-14T19:41:50.3206909Z", "durationInMs": 0.2102},{"event": "Transit Time", "startTimeUtc": "2026-04-14T19:41:50.3209011Z", "durationInMs": 1.8952},{"event": "Received", "startTimeUtc": "2026-04-14T19:41:50.3227963Z", "durationInMs": 0.1504},{"event": "Completed", "startTimeUtc": "2026-04-14T19:41:50.3229467Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T19:41:50.3189595Z","lastSend":"2026-04-14T19:41:50.3189595Z","lastReceive":"2026-04-14T19:41:50.3189595Z"},"requestSizeInBytes":1598,"requestBodySizeInBytes":1146,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:41:50.2877169Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"c37b1dfb-3e43-4c57-9485-e13ea5fde11a","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T19:41:50.2877169Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"6716d625-074d-4904-90d4-6d8443a1549c","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T19:41:50.2877169Z","DurationInMs":15.6134,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"ac2a84f6-cd36-4601-999d-20289baadeb7","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.343759,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"ac2a84f6-cd36-4601-999d-20289baadeb7","ResponseTimeUtc":"2026-04-14T19:41:50.3189595Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: ac2a84f6-cd36-4601-999d-20289baadeb7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150125","BELatencyInMs":"0.607"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 19:54:01 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 1c4bd08f-ae4f-4e47-b69b-6525ebc7189c; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 1c4bd08f-ae4f-4e47-b69b-6525ebc7189c, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 1c4bd08f-ae4f-4e47-b69b-6525ebc7189c; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 1c4bd08f-ae4f-4e47-b69b-6525ebc7189c, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T19:54:01.472Z","duration in milliseconds":47.4294,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T19:54:00.6289392Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93505,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0466},{"name":"Get PkValue From Stream","duration in milliseconds":7.7986,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":7.6345,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:54:01.4726992Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"14d2edcf-1196-4e82-a81c-966b6a9a53ce","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":39.3969,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":39.3613,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T19:53:03.6967181Z","cpu":2.028,"memory":5271720.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1047,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:53:13.7120905Z","cpu":2.730,"memory":5267100.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0984,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:53:23.7138571Z","cpu":2.891,"memory":5261952.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1045,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:53:33.7217499Z","cpu":1.797,"memory":5260104.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.437,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:53:43.7322027Z","cpu":1.404,"memory":5259072.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1081,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:53:53.7335234Z","cpu":1.641,"memory":5269276.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1137,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":39.3543,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":39.3509,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":39.328,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":39.3248,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":38.357600000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T19:54:01.4883354Z","EndTimeUTC":"2026-04-14T19:54:01.4883354Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T19:54:01.5195756Z","DurationInMs":31.2402,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"1c4bd08f-ae4f-4e47-b69b-6525ebc7189c","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150163,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150163,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150163,"SessionToken":"-1#26150163","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.647","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 7:54:01 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T19:54:01.4883354Z", "durationInMs": 0.0079},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T19:54:01.4883433Z", "durationInMs": 20.0642},{"event": "Pipelined", "startTimeUtc": "2026-04-14T19:54:01.5084075Z", "durationInMs": 0.2031},{"event": "Transit Time", "startTimeUtc": "2026-04-14T19:54:01.5086106Z", "durationInMs": 2.1035},{"event": "Received", "startTimeUtc": "2026-04-14T19:54:01.5107141Z", "durationInMs": 0.1494},{"event": "Completed", "startTimeUtc": "2026-04-14T19:54:01.5108635Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T19:54:01.5039489Z","lastSend":"2026-04-14T19:54:01.5039489Z","lastReceive":"2026-04-14T19:54:01.5195756Z"},"requestSizeInBytes":1592,"requestBodySizeInBytes":1144,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:54:01.4726992Z","DurationInMs":15.6362,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0f19a1a6-fd2a-4813-89c4-98cbbd65bc37","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T19:54:01.4883354Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"052f53f5-3261-4fbc-bdd5-79a6a998f0bf","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T19:54:01.4883354Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"1c4bd08f-ae4f-4e47-b69b-6525ebc7189c","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.8906364,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"1c4bd08f-ae4f-4e47-b69b-6525ebc7189c","ResponseTimeUtc":"2026-04-14T19:54:01.5195756Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 1c4bd08f-ae4f-4e47-b69b-6525ebc7189c, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150163","BELatencyInMs":"0.647"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 19:54:48 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e2531d65-b502-4251-85aa-5f384abbb9ea; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e2531d65-b502-4251-85aa-5f384abbb9ea, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e2531d65-b502-4251-85aa-5f384abbb9ea; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e2531d65-b502-4251-85aa-5f384abbb9ea, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T19:54:48.580Z","duration in milliseconds":48.3333,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T19:54:48.0956512Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93515,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.05},{"name":"Get PkValue From Stream","duration in milliseconds":6.8709000000000007,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0013000000000000002},{"name":"Read Collection","duration in milliseconds":6.7082000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:54:48.5800325Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"2eff5a83-2e7e-46b5-9f35-bc8ce2cdc319","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":41.237100000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":41.2006,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T19:53:53.7335234Z","cpu":1.641,"memory":5269276.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1137,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:54:03.7478036Z","cpu":3.900,"memory":5265668.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0719,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:54:13.7499647Z","cpu":3.359,"memory":5264620.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1231,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:54:23.7571695Z","cpu":2.106,"memory":5270952.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.093,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:54:33.7729075Z","cpu":2.886,"memory":5267196.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1223,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:54:43.7848839Z","cpu":1.563,"memory":5262264.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1103,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":41.1882,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":41.183800000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":41.163000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":41.1606,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":40.304100000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T19:54:48.5957275Z","EndTimeUTC":"2026-04-14T19:54:48.5957275Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T19:54:48.6269100Z","DurationInMs":31.1825,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"e2531d65-b502-4251-85aa-5f384abbb9ea","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150165,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150165,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150165,"SessionToken":"-1#26150165","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.587","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 7:54:48 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T19:54:48.5957275Z", "durationInMs": 0.0086},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T19:54:48.5957361Z", "durationInMs": 18.6749},{"event": "Pipelined", "startTimeUtc": "2026-04-14T19:54:48.6144110Z", "durationInMs": 0.2195},{"event": "Transit Time", "startTimeUtc": "2026-04-14T19:54:48.6146305Z", "durationInMs": 5.2828},{"event": "Received", "startTimeUtc": "2026-04-14T19:54:48.6199133Z", "durationInMs": 0.1202},{"event": "Completed", "startTimeUtc": "2026-04-14T19:54:48.6200335Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T19:54:48.6114027Z","lastSend":"2026-04-14T19:54:48.6114027Z","lastReceive":"2026-04-14T19:54:48.6114027Z"},"requestSizeInBytes":1604,"requestBodySizeInBytes":1156,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T19:54:48.5800325Z","DurationInMs":15.695,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"4d585713-a6c8-4fa7-958e-652e64e2ac67","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T19:54:48.5957275Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"522b6253-ab20-474f-b92b-6c05379cb419","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T19:54:48.5957275Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"e2531d65-b502-4251-85aa-5f384abbb9ea","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.5312588,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"e2531d65-b502-4251-85aa-5f384abbb9ea","ResponseTimeUtc":"2026-04-14T19:54:48.6269100Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: e2531d65-b502-4251-85aa-5f384abbb9ea, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150165","BELatencyInMs":"0.587"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0014}]}
2026-04-14 20:00:29 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 68715f11-fd7b-4572-b3c3-c5b720309c1c; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 68715f11-fd7b-4572-b3c3-c5b720309c1c, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 68715f11-fd7b-4572-b3c3-c5b720309c1c; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 68715f11-fd7b-4572-b3c3-c5b720309c1c, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:00:29.246Z","duration in milliseconds":49.696400000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:00:28.5585109Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93579,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.046900000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":6.521,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0019},{"name":"Read Collection","duration in milliseconds":6.3647,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:00:29.2460260Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"9813ee81-22fe-4d65-83c2-0d8d2817b5f4","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":42.9485,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":42.915600000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T19:59:34.0430376Z","cpu":1.638,"memory":5317788.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0884,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:59:44.0458626Z","cpu":0.469,"memory":5307876.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0723,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T19:59:54.0526139Z","cpu":1.563,"memory":5314568.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0884,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:00:04.0599987Z","cpu":9.672,"memory":5295852.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0484,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:00:14.0685564Z","cpu":3.438,"memory":5239712.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0992,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:00:24.0764850Z","cpu":4.290,"memory":5239868.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0944,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":42.9084,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":42.903,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":42.8787,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":42.8765,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":41.909600000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:00:29.2617150Z","EndTimeUTC":"2026-04-14T20:00:29.2617150Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:00:29.2929003Z","DurationInMs":31.1853,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"68715f11-fd7b-4572-b3c3-c5b720309c1c","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150181,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150181,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150181,"SessionToken":"-1#26150181","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.695","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:00:29 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:00:29.2617150Z", "durationInMs": 0.0088},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:00:29.2617238Z", "durationInMs": 19.3408},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:00:29.2810646Z", "durationInMs": 0.2315},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:00:29.2812961Z", "durationInMs": 2.0349},{"event": "Received", "startTimeUtc": "2026-04-14T20:00:29.2833310Z", "durationInMs": 0.1505},{"event": "Completed", "startTimeUtc": "2026-04-14T20:00:29.2834815Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:00:29.2929003Z","lastSend":"2026-04-14T20:00:29.2929003Z","lastReceive":"2026-04-14T20:00:29.2929003Z"},"requestSizeInBytes":1608,"requestBodySizeInBytes":1156,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:00:29.2460260Z","DurationInMs":15.689,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"79b98aa1-80fd-43e8-826f-c7d39552da30","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:00:29.2617150Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0be9ae61-0e8c-4b35-a33d-fb0708c1d38f","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:00:29.2617150Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"68715f11-fd7b-4572-b3c3-c5b720309c1c","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.7343894,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"68715f11-fd7b-4572-b3c3-c5b720309c1c","ResponseTimeUtc":"2026-04-14T20:00:29.2929003Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 68715f11-fd7b-4572-b3c3-c5b720309c1c, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150181","BELatencyInMs":"0.695"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 20:00:55 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 911ae397-9b82-49b5-b834-b9443410dbdd; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 911ae397-9b82-49b5-b834-b9443410dbdd, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 911ae397-9b82-49b5-b834-b9443410dbdd; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 911ae397-9b82-49b5-b834-b9443410dbdd, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:00:55.181Z","duration in milliseconds":48.3158,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:00:54.5100006Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93585,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0505},{"name":"Get PkValue From Stream","duration in milliseconds":6.8598,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":6.7046,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:00:55.1818871Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"3b56dec9-c24d-4ea6-9de3-1ff66dd1d852","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":41.2138,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":41.1784,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:00:04.0599987Z","cpu":9.672,"memory":5295852.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0484,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:00:14.0685564Z","cpu":3.438,"memory":5239712.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0992,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:00:24.0764850Z","cpu":4.290,"memory":5239868.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0944,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:00:34.0826334Z","cpu":2.578,"memory":5235976.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1138,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:00:44.0964827Z","cpu":2.418,"memory":5235844.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1208,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:00:54.1037614Z","cpu":1.641,"memory":5234216.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1304,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":41.1666,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":41.1605,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":41.1402,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":41.1377,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":40.2537,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:00:55.1975830Z","EndTimeUTC":"2026-04-14T20:00:55.1975830Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:00:55.2287767Z","DurationInMs":15.632200000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"911ae397-9b82-49b5-b834-b9443410dbdd","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150182,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150182,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150182,"SessionToken":"-1#26150182","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.567","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:00:55 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:00:55.2131445Z", "durationInMs": 0.0112},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:00:55.2131557Z", "durationInMs": 18.7261},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:00:55.2318818Z", "durationInMs": 0.2051},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:00:55.2320869Z", "durationInMs": 2.0732},{"event": "Received", "startTimeUtc": "2026-04-14T20:00:55.2341601Z", "durationInMs": 0.1469},{"event": "Completed", "startTimeUtc": "2026-04-14T20:00:55.2343070Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:00:55.2287767Z","lastSend":"2026-04-14T20:00:55.2287767Z","lastReceive":"2026-04-14T20:00:55.2287767Z"},"requestSizeInBytes":1602,"requestBodySizeInBytes":1148,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:00:55.1818871Z","DurationInMs":15.6959,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"d3466a30-b951-49cc-915b-5bf016521425","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:00:55.1975830Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"570bb9d0-5465-47d6-ac8e-6042fcd8143e","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:00:55.1975830Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"911ae397-9b82-49b5-b834-b9443410dbdd","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.71877609999999992,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"911ae397-9b82-49b5-b834-b9443410dbdd","ResponseTimeUtc":"2026-04-14T20:00:55.2287767Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 911ae397-9b82-49b5-b834-b9443410dbdd, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150182","BELatencyInMs":"0.567"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 20:03:34 | Log4Net | Unhandled exception in DiscardArchivalMaterials: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c53276a7-e5a6-458f-9c3f-d120b5dcf6d9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c53276a7-e5a6-458f-9c3f-d120b5dcf6d9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c53276a7-e5a6-458f-9c3f-d120b5dcf6d9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c53276a7-e5a6-458f-9c3f-d120b5dcf6d9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:03:34.268Z","duration in milliseconds":4.3827,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:03:33.4563556Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93630,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0409},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":4.1987000000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":4.1614,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:02:44.2263707Z","cpu":4.134,"memory":5269156.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1177,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:02:54.2384072Z","cpu":5.226,"memory":5267584.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0652,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:03:04.2473248Z","cpu":2.422,"memory":5319452.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1171,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:03:14.2550021Z","cpu":5.148,"memory":5309684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0903,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:03:24.2604137Z","cpu":3.750,"memory":5305892.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0949,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:03:34.2688878Z","cpu":3.042,"memory":5303784.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0933,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":4}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":4.1566,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":4.1531,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":4.1349,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":4.1328000000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":3.1390000000000002,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:03:34.2688878Z","DurationInMs":0,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c53276a7-e5a6-458f-9c3f-d120b5dcf6d9","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150193,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150193,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150193,"SessionToken":"-1#26150193","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.572","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:03:34 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:03:34.2688878Z", "durationInMs": 0.0034},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:03:34.2688912Z", "durationInMs": 0.0051},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:03:34.2688963Z", "durationInMs": 0.2873},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:03:34.2691836Z", "durationInMs": 1.9609},{"event": "Received", "startTimeUtc": "2026-04-14T20:03:34.2711445Z", "durationInMs": 0.0693},{"event": "Completed", "startTimeUtc": "2026-04-14T20:03:34.2712138Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:03:34.2376193Z","lastSend":"2026-04-14T20:03:34.2376193Z","lastReceive":"2026-04-14T20:03:34.2688878Z"},"requestSizeInBytes":1562,"requestBodySizeInBytes":1114,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.81253219999999993,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c53276a7-e5a6-458f-9c3f-d120b5dcf6d9","ResponseTimeUtc":"2026-04-14T20:03:34.2688878Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c53276a7-e5a6-458f-9c3f-d120b5dcf6d9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150193","BELatencyInMs":"0.572"}}}]}]}]}]}]}]}]}
2026-04-14 20:06:34 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b9603173-7b93-4648-b98a-ef8a7729d626; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b9603173-7b93-4648-b98a-ef8a7729d626, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b9603173-7b93-4648-b98a-ef8a7729d626; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b9603173-7b93-4648-b98a-ef8a7729d626, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:06:34.778Z","duration in milliseconds":43.4707,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:06:34.6539217Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93654,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.044500000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":6.3069000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003},{"name":"Read Collection","duration in milliseconds":6.1189,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:06:34.7789221Z","DurationInMs":15.6279,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"c032be48-d48d-43f6-bddb-b6a5fa2471ce","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.9339,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.9001,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:05:44.3878407Z","cpu":0.859,"memory":5279156.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2128,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:05:54.3935664Z","cpu":3.354,"memory":5292264.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1258,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:06:04.3991152Z","cpu":5.703,"memory":5234768.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0992,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:06:14.4136453Z","cpu":3.042,"memory":5235104.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1104,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:06:24.4148967Z","cpu":0.859,"memory":5240708.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1158,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:06:34.4196189Z","cpu":1.484,"memory":5235888.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1209,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.89,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.885200000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.864200000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.8619,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.9255,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:06:34.7945500Z","EndTimeUTC":"2026-04-14T20:06:34.8101731Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:06:34.8258525Z","DurationInMs":15.679400000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"b9603173-7b93-4648-b98a-ef8a7729d626","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150201,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150201,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150201,"SessionToken":"-1#26150201","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.756","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:06:34 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:06:34.8101731Z", "durationInMs": 0.0101},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:06:34.8101832Z", "durationInMs": 18.7665},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:06:34.8289497Z", "durationInMs": 0.1909},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:06:34.8291406Z", "durationInMs": 2.2556},{"event": "Received", "startTimeUtc": "2026-04-14T20:06:34.8313962Z", "durationInMs": 0.1279},{"event": "Completed", "startTimeUtc": "2026-04-14T20:06:34.8315241Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:06:34.8258525Z","lastSend":"2026-04-14T20:06:34.8258525Z","lastReceive":"2026-04-14T20:06:34.8258525Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1186,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:06:34.7945500Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"fe49e5e2-331b-4300-8c29-d06e59973faf","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:06:34.7945500Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"39e5d1a9-d60a-4f2a-9e50-ec6d8993bb80","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:06:34.7945500Z","DurationInMs":15.6231,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"b9603173-7b93-4648-b98a-ef8a7729d626","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1719308,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"b9603173-7b93-4648-b98a-ef8a7729d626","ResponseTimeUtc":"2026-04-14T20:06:34.8258525Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: b9603173-7b93-4648-b98a-ef8a7729d626, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150201","BELatencyInMs":"0.756"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 20:06:52 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 09bcb450-c88b-4aba-923e-4c435b0cfca8; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 09bcb450-c88b-4aba-923e-4c435b0cfca8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 09bcb450-c88b-4aba-923e-4c435b0cfca8; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 09bcb450-c88b-4aba-923e-4c435b0cfca8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:06:52.121Z","duration in milliseconds":40.749,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:06:52.0277150Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93658,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0533},{"name":"Get PkValue From Stream","duration in milliseconds":5.9179,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0046},{"name":"Read Collection","duration in milliseconds":5.7521,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:06:52.1215390Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"ac104e7a-fc9f-4d4a-aa05-7c7fe9a511e2","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":34.598600000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":34.563900000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:05:54.3935664Z","cpu":3.354,"memory":5292264.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1258,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:06:04.3991152Z","cpu":5.703,"memory":5234768.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0992,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:06:14.4136453Z","cpu":3.042,"memory":5235104.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1104,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:06:24.4148967Z","cpu":0.859,"memory":5240708.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1158,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:06:34.4196189Z","cpu":1.484,"memory":5235888.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1209,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:06:44.4271620Z","cpu":2.730,"memory":5238404.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0989,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":34.5572,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":34.5523,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":34.5341,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":34.5308,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":33.6469,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:06:52.1372177Z","EndTimeUTC":"2026-04-14T20:06:52.1372177Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:06:52.1527179Z","DurationInMs":15.500200000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"09bcb450-c88b-4aba-923e-4c435b0cfca8","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150201,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150201,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150201,"SessionToken":"-1#26150201","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.543","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:06:52 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:06:52.1372177Z", "durationInMs": 0.0095},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:06:52.1372272Z", "durationInMs": 16.9093},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:06:52.1541365Z", "durationInMs": 0.2071},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:06:52.1543436Z", "durationInMs": 1.8396},{"event": "Received", "startTimeUtc": "2026-04-14T20:06:52.1561832Z", "durationInMs": 0.1381},{"event": "Completed", "startTimeUtc": "2026-04-14T20:06:52.1563213Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:06:52.1527179Z","lastSend":"2026-04-14T20:06:52.1527179Z","lastReceive":"2026-04-14T20:06:52.1527179Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:06:52.1215390Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"440b5ae6-e404-49db-b2d3-44f3621c08e7","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:06:52.1215390Z","DurationInMs":15.678700000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"aa4f5098-a114-40c2-b710-4fe3220b8865","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:06:52.1372177Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"09bcb450-c88b-4aba-923e-4c435b0cfca8","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1250029,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"09bcb450-c88b-4aba-923e-4c435b0cfca8","ResponseTimeUtc":"2026-04-14T20:06:52.1527179Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 09bcb450-c88b-4aba-923e-4c435b0cfca8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150201","BELatencyInMs":"0.543"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.003}]}
2026-04-14 20:07:26 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: cdcb8b3a-58fc-4dca-9658-55ffdcf89593; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: cdcb8b3a-58fc-4dca-9658-55ffdcf89593, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: cdcb8b3a-58fc-4dca-9658-55ffdcf89593; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: cdcb8b3a-58fc-4dca-9658-55ffdcf89593, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:07:26.692Z","duration in milliseconds":44.2808,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:07:26.6146504Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93664,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.053700000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":6.6858,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001},{"name":"Read Collection","duration in milliseconds":6.5092,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:07:26.6927910Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"a59eaec8-b9e8-4afd-b030-8f7aae6ab22e","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.2676,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.228,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:06:34.4196189Z","cpu":1.484,"memory":5235888.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1209,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:06:44.4271620Z","cpu":2.730,"memory":5238404.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0989,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:06:54.4327556Z","cpu":1.406,"memory":5238808.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0686,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:07:04.4466605Z","cpu":1.794,"memory":5238224.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1416,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:07:14.4590352Z","cpu":3.276,"memory":5239404.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1013,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:07:24.4694490Z","cpu":2.891,"memory":5239540.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1033,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.22,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.216100000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.1929,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.1899,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.2004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:07:26.7084024Z","EndTimeUTC":"2026-04-14T20:07:26.7084024Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:07:26.7240284Z","DurationInMs":15.626000000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"cdcb8b3a-58fc-4dca-9658-55ffdcf89593","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150203,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150203,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150203,"SessionToken":"-1#26150203","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.63","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:07:26 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:07:26.7084024Z", "durationInMs": 0.0087},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:07:26.7084111Z", "durationInMs": 17.7067},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:07:26.7261178Z", "durationInMs": 0.5697},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:07:26.7266875Z", "durationInMs": 1.8437},{"event": "Received", "startTimeUtc": "2026-04-14T20:07:26.7285312Z", "durationInMs": 0.1424},{"event": "Completed", "startTimeUtc": "2026-04-14T20:07:26.7286736Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:07:26.7240284Z","lastSend":"2026-04-14T20:07:26.7240284Z","lastReceive":"2026-04-14T20:07:26.7240284Z"},"requestSizeInBytes":1633,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:07:26.6927910Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"5fc0a5a9-da32-4342-99d6-d815a0d0879d","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:07:26.6927910Z","DurationInMs":15.611400000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"477291be-b193-4821-ba5d-39b584f93f7b","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:07:26.7084024Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"cdcb8b3a-58fc-4dca-9658-55ffdcf89593","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.10937799999999999,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"cdcb8b3a-58fc-4dca-9658-55ffdcf89593","ResponseTimeUtc":"2026-04-14T20:07:26.7240284Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: cdcb8b3a-58fc-4dca-9658-55ffdcf89593, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150203","BELatencyInMs":"0.63"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0024000000000000002}]}
2026-04-14 20:08:19 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 3d46974b-e908-4b4a-85b1-6ce188e6ceaf; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 3d46974b-e908-4b4a-85b1-6ce188e6ceaf, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 3d46974b-e908-4b4a-85b1-6ce188e6ceaf; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 3d46974b-e908-4b4a-85b1-6ce188e6ceaf, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:08:19.782Z","duration in milliseconds":55.5047,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:08:19.6256670Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93672,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0529},{"name":"Get PkValue From Stream","duration in milliseconds":6.7537,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0024000000000000002},{"name":"Read Collection","duration in milliseconds":6.5868,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:08:19.7820014Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"dab2800c-52ec-4c94-ac69-88195cde4ec4","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":48.5186,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":48.4799,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:07:24.4694490Z","cpu":2.891,"memory":5239540.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1033,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:07:34.4749862Z","cpu":0.858,"memory":5238920.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0981,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:07:44.4821144Z","cpu":1.484,"memory":5238264.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1144,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:07:54.4916523Z","cpu":1.092,"memory":5254984.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0996,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:08:04.4929167Z","cpu":2.969,"memory":5306536.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0998,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:08:14.5016454Z","cpu":2.656,"memory":5304224.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1005,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":48.4727,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":48.4683,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":48.444300000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":48.441700000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":47.5604,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:08:19.7976032Z","EndTimeUTC":"2026-04-14T20:08:19.7976032Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:08:19.8287951Z","DurationInMs":31.1919,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"3d46974b-e908-4b4a-85b1-6ce188e6ceaf","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150206,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150206,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150206,"SessionToken":"-1#26150206","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.574","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:08:19 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:08:19.7976032Z", "durationInMs": 0.0096},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:08:19.7976128Z", "durationInMs": 27.3174},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:08:19.8249302Z", "durationInMs": 0.2043},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:08:19.8251345Z", "durationInMs": 1.8193},{"event": "Received", "startTimeUtc": "2026-04-14T20:08:19.8269538Z", "durationInMs": 0.1515},{"event": "Completed", "startTimeUtc": "2026-04-14T20:08:19.8271053Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:08:19.8287951Z","lastSend":"2026-04-14T20:08:19.8287951Z","lastReceive":"2026-04-14T20:08:19.8287951Z"},"requestSizeInBytes":1630,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:08:19.7820014Z","DurationInMs":15.6018,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a9937e1f-30a8-4dde-ba62-3a1643488d94","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:08:19.7976032Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0a26d645-2511-475c-9229-d87651023582","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:08:19.7976032Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"3d46974b-e908-4b4a-85b1-6ce188e6ceaf","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.20312809999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"3d46974b-e908-4b4a-85b1-6ce188e6ceaf","ResponseTimeUtc":"2026-04-14T20:08:19.8287951Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 3d46974b-e908-4b4a-85b1-6ce188e6ceaf, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150206","BELatencyInMs":"0.574"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 20:10:17 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 4396a390-ea9a-4217-9006-2f5d6606f785; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 4396a390-ea9a-4217-9006-2f5d6606f785, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 4396a390-ea9a-4217-9006-2f5d6606f785; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 4396a390-ea9a-4217-9006-2f5d6606f785, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:10:17.947Z","duration in milliseconds":50.222,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:10:17.2599788Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93696,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0449},{"name":"Get PkValue From Stream","duration in milliseconds":6.4956000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.3458000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:10:17.9474834Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"0700f911-efee-4f13-803a-cf28e3f79444","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":43.5095,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":43.4772,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:09:24.5605604Z","cpu":1.170,"memory":5288432.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0324,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:09:34.5614289Z","cpu":0.625,"memory":5286296.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0986,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:09:44.5669943Z","cpu":2.031,"memory":5280564.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1062,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:09:54.5752875Z","cpu":3.042,"memory":5281456.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1017,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:10:04.5869301Z","cpu":3.120,"memory":5282684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0998,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:10:14.5938370Z","cpu":5.703,"memory":5217056.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.097,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":43.4701,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":43.4658,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":43.446200000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":43.4433,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":42.4803,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:10:17.9632058Z","EndTimeUTC":"2026-04-14T20:10:17.9788142Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:10:17.9944277Z","DurationInMs":15.6135,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"4396a390-ea9a-4217-9006-2f5d6606f785","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150215,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150215,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150215,"SessionToken":"-1#26150215","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.862","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:10:17 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:10:17.9788142Z", "durationInMs": 0.0095},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:10:17.9788237Z", "durationInMs": 17.613},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:10:17.9964367Z", "durationInMs": 0.2192},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:10:17.9966559Z", "durationInMs": 2.0869},{"event": "Received", "startTimeUtc": "2026-04-14T20:10:17.9987428Z", "durationInMs": 0.1478},{"event": "Completed", "startTimeUtc": "2026-04-14T20:10:17.9988906Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:10:17.9944277Z","lastSend":"2026-04-14T20:10:17.9944277Z","lastReceive":"2026-04-14T20:10:17.9944277Z"},"requestSizeInBytes":1604,"requestBodySizeInBytes":1154,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:10:17.9474834Z","DurationInMs":15.7224,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"f559156a-8e3e-4cca-91fa-136c35ec0708","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:10:17.9632058Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"f50c15ee-b01c-4df1-8e83-bfb9737ae475","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:10:17.9632058Z","DurationInMs":15.608400000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"4396a390-ea9a-4217-9006-2f5d6606f785","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.7344489,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"4396a390-ea9a-4217-9006-2f5d6606f785","ResponseTimeUtc":"2026-04-14T20:10:17.9944277Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 4396a390-ea9a-4217-9006-2f5d6606f785, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150215","BELatencyInMs":"0.862"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.004}]}
2026-04-14 20:10:55 | Log4Net | Unhandled exception in DiscardArchivalMaterials: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 28958385-9119-45d9-8a19-338158a8ee64; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 28958385-9119-45d9-8a19-338158a8ee64, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 28958385-9119-45d9-8a19-338158a8ee64; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 28958385-9119-45d9-8a19-338158a8ee64, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:10:55.394Z","duration in milliseconds":4.1772,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:10:54.5712560Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93704,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.056100000000000004},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":3.9674,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":3.9253,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:10:04.5869301Z","cpu":3.120,"memory":5282684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0998,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:10:14.5938370Z","cpu":5.703,"memory":5217056.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.097,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:10:24.6012145Z","cpu":2.808,"memory":5216144.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3003,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:10:34.6053300Z","cpu":3.906,"memory":5220104.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1054,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:10:44.6176964Z","cpu":1.404,"memory":5221944.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0736,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:10:54.6337667Z","cpu":1.560,"memory":5221920.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1007,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":3.919,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":3.9141000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":3.8956000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":3.8915,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":2.9930000000000003,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:10:55.3940999Z","DurationInMs":0,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"28958385-9119-45d9-8a19-338158a8ee64","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150217,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150217,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150217,"SessionToken":"-1#26150217","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.528","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:10:55 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:10:55.3940999Z", "durationInMs": 0.0041},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:10:55.3941040Z", "durationInMs": 0.0054},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:10:55.3941094Z", "durationInMs": 0.2243},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:10:55.3943337Z", "durationInMs": 1.8857},{"event": "Received", "startTimeUtc": "2026-04-14T20:10:55.3962194Z", "durationInMs": 0.0753},{"event": "Completed", "startTimeUtc": "2026-04-14T20:10:55.3962947Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:10:55.3784758Z","lastSend":"2026-04-14T20:10:55.3784758Z","lastReceive":"2026-04-14T20:10:55.3940999Z"},"requestSizeInBytes":1564,"requestBodySizeInBytes":1114,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.8228439,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"28958385-9119-45d9-8a19-338158a8ee64","ResponseTimeUtc":"2026-04-14T20:10:55.3940999Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 28958385-9119-45d9-8a19-338158a8ee64, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150217","BELatencyInMs":"0.528"}}}]}]}]}]}]}]}]}
2026-04-14 20:13:49 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 8eb2e6de-3702-434e-95b5-7c83bba9d8fb; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 8eb2e6de-3702-434e-95b5-7c83bba9d8fb, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 8eb2e6de-3702-434e-95b5-7c83bba9d8fb; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 8eb2e6de-3702-434e-95b5-7c83bba9d8fb, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:13:49.576Z","duration in milliseconds":47.2916,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:13:49.0763582Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93740,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0396},{"name":"Get PkValue From Stream","duration in milliseconds":6.7389,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.5938,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:13:49.5764499Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"410a0530-7fb8-4e8b-bada-309110786db2","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":40.3472,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":40.3173,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:12:54.7454951Z","cpu":4.212,"memory":5237864.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1198,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:13:04.7523865Z","cpu":3.594,"memory":5293488.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0721,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:13:14.7553996Z","cpu":3.828,"memory":5287708.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1273,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:13:24.7639912Z","cpu":3.120,"memory":5281572.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1386,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:13:34.7686381Z","cpu":1.953,"memory":5279728.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0478,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:13:44.7827670Z","cpu":2.340,"memory":5367000.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1191,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":40.310900000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":40.307300000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":40.2884,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":40.2849,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":39.3348,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:13:49.5919899Z","EndTimeUTC":"2026-04-14T20:13:49.5919899Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:13:49.6232372Z","DurationInMs":31.247300000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"8eb2e6de-3702-434e-95b5-7c83bba9d8fb","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150235,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150235,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150235,"SessionToken":"-1#26150235","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.635","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:13:49 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:13:49.5919899Z", "durationInMs": 0.0095},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:13:49.5919994Z", "durationInMs": 21.7803},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:13:49.6137797Z", "durationInMs": 0.2251},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:13:49.6140048Z", "durationInMs": 2.0798},{"event": "Received", "startTimeUtc": "2026-04-14T20:13:49.6160846Z", "durationInMs": 0.1399},{"event": "Completed", "startTimeUtc": "2026-04-14T20:13:49.6162245Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:13:49.6076138Z","lastSend":"2026-04-14T20:13:49.6076138Z","lastReceive":"2026-04-14T20:13:49.6076138Z"},"requestSizeInBytes":1602,"requestBodySizeInBytes":1154,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:13:49.5764499Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"9518185b-7952-4ada-8399-598af6bbb6a4","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:13:49.5764499Z","DurationInMs":15.540000000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"7774f881-f8d9-4d8d-8bff-32fe52f7bd20","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:13:49.5919899Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"8eb2e6de-3702-434e-95b5-7c83bba9d8fb","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.546879,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"8eb2e6de-3702-434e-95b5-7c83bba9d8fb","ResponseTimeUtc":"2026-04-14T20:13:49.6232372Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 8eb2e6de-3702-434e-95b5-7c83bba9d8fb, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150235","BELatencyInMs":"0.635"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 20:18:04 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6180fe6a-4b20-4cd8-b9ad-add4f925d18f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6180fe6a-4b20-4cd8-b9ad-add4f925d18f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6180fe6a-4b20-4cd8-b9ad-add4f925d18f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6180fe6a-4b20-4cd8-b9ad-add4f925d18f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:18:04.235Z","duration in milliseconds":44.197900000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:18:04.0948837Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93773,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0512},{"name":"Get PkValue From Stream","duration in milliseconds":6.847,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.002},{"name":"Read Collection","duration in milliseconds":6.6841,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:18:04.2355139Z","DurationInMs":15.624500000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"de66573b-bd7a-4c59-9670-8cdfec252411","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.1223,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.0893,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:17:04.9433737Z","cpu":1.719,"memory":5256084.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1243,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:17:14.9459601Z","cpu":0.703,"memory":5255644.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0996,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:17:24.9614809Z","cpu":2.808,"memory":5173184.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0996,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:17:34.9727263Z","cpu":2.184,"memory":5177632.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0996,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:17:44.9761797Z","cpu":3.125,"memory":5183912.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1017,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:17:54.9809832Z","cpu":3.125,"memory":5188676.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0773,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.082300000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.0769,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.0576,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.0554,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.0861,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:18:04.2511384Z","EndTimeUTC":"2026-04-14T20:18:04.2667607Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:18:04.2823889Z","DurationInMs":15.628200000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"6180fe6a-4b20-4cd8-b9ad-add4f925d18f","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150249,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150249,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150249,"SessionToken":"-1#26150249","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.705","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:18:04 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:18:04.2667607Z", "durationInMs": 0.0097},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:18:04.2667704Z", "durationInMs": 19.8797},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:18:04.2866501Z", "durationInMs": 0.2327},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:18:04.2868828Z", "durationInMs": 2.0523},{"event": "Received", "startTimeUtc": "2026-04-14T20:18:04.2889351Z", "durationInMs": 0.1367},{"event": "Completed", "startTimeUtc": "2026-04-14T20:18:04.2890718Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:18:04.2823889Z","lastSend":"2026-04-14T20:18:04.2823889Z","lastReceive":"2026-04-14T20:18:04.2823889Z"},"requestSizeInBytes":1637,"requestBodySizeInBytes":1183,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:18:04.2511384Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"6daacb92-5480-4b79-be8e-70516eef4cf0","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:18:04.2511384Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"2534580d-a4b4-4088-aa1a-59b28b4e8b16","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:18:04.2511384Z","DurationInMs":15.622300000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"6180fe6a-4b20-4cd8-b9ad-add4f925d18f","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.18750519999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"6180fe6a-4b20-4cd8-b9ad-add4f925d18f","ResponseTimeUtc":"2026-04-14T20:18:04.2823889Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 6180fe6a-4b20-4cd8-b9ad-add4f925d18f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150249","BELatencyInMs":"0.705"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001}]}
2026-04-14 20:18:22 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c17cf068-6f9e-4a7e-9e82-0796ca9972cf; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c17cf068-6f9e-4a7e-9e82-0796ca9972cf, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c17cf068-6f9e-4a7e-9e82-0796ca9972cf; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c17cf068-6f9e-4a7e-9e82-0796ca9972cf, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:18:22.551Z","duration in milliseconds":53.224700000000006,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:18:22.4420244Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93777,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.041800000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":6.9513000000000007,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.8161000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:18:22.5513972Z","DurationInMs":15.6278,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"5da5f66f-870d-4845-81f0-7c7a7ca1ed33","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":46.0664,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":46.0407,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:17:24.9614809Z","cpu":2.808,"memory":5173184.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0996,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:17:34.9727263Z","cpu":2.184,"memory":5177632.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0996,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:17:44.9761797Z","cpu":3.125,"memory":5183912.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1017,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:17:54.9809832Z","cpu":3.125,"memory":5188676.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0773,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:18:04.9923152Z","cpu":6.006,"memory":5329288.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0415,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:18:15.0076092Z","cpu":3.744,"memory":5325304.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1127,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":46.033300000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":46.0279,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":46.0096,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":46.007400000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":45.186,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:18:22.5670250Z","EndTimeUTC":"2026-04-14T20:18:22.5670250Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:18:22.5982748Z","DurationInMs":31.2498,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c17cf068-6f9e-4a7e-9e82-0796ca9972cf","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150251,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150251,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150251,"SessionToken":"-1#26150251","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.559","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:18:22 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:18:22.5670250Z", "durationInMs": 0.0085},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:18:22.5670335Z", "durationInMs": 29.314},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:18:22.5963475Z", "durationInMs": 0.2067},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:18:22.5965542Z", "durationInMs": 1.8343},{"event": "Received", "startTimeUtc": "2026-04-14T20:18:22.5983885Z", "durationInMs": 0.1382},{"event": "Completed", "startTimeUtc": "2026-04-14T20:18:22.5985267Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:18:22.5982748Z","lastSend":"2026-04-14T20:18:22.5982748Z","lastReceive":"2026-04-14T20:18:22.5982748Z"},"requestSizeInBytes":1635,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:18:22.5670250Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"56d48fc8-02e4-4450-a3bb-3316608dc62c","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:18:22.5670250Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"95f63329-bbd2-4c0b-8497-9e07f53d1fa5","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:18:22.5670250Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"c17cf068-6f9e-4a7e-9e82-0796ca9972cf","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718737,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c17cf068-6f9e-4a7e-9e82-0796ca9972cf","ResponseTimeUtc":"2026-04-14T20:18:22.6138981Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c17cf068-6f9e-4a7e-9e82-0796ca9972cf, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150251","BELatencyInMs":"0.559"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 20:18:36 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c201ca2a-836f-4fa0-9884-a7086ac41bdf; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c201ca2a-836f-4fa0-9884-a7086ac41bdf, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c201ca2a-836f-4fa0-9884-a7086ac41bdf; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c201ca2a-836f-4fa0-9884-a7086ac41bdf, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:18:36.141Z","duration in milliseconds":53.351800000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:18:36.0167822Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93783,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0494},{"name":"Get PkValue From Stream","duration in milliseconds":6.7249000000000008,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.5608,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:18:36.1417898Z","DurationInMs":15.681700000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"9db9482e-3acb-4230-b517-95d6528cd89a","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":46.398500000000006,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":46.3669,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:17:44.9761797Z","cpu":3.125,"memory":5183912.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1017,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:17:54.9809832Z","cpu":3.125,"memory":5188676.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0773,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:18:04.9923152Z","cpu":6.006,"memory":5329288.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0415,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:18:15.0076092Z","cpu":3.744,"memory":5325304.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1127,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:18:25.0202725Z","cpu":3.822,"memory":5339500.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1123,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:18:35.0324504Z","cpu":5.382,"memory":5334580.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0995,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":46.357600000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":46.3528,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":46.3314,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":46.3278,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":45.4525,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:18:36.1574715Z","EndTimeUTC":"2026-04-14T20:18:36.1730349Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:18:36.2042880Z","DurationInMs":31.2531,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c201ca2a-836f-4fa0-9884-a7086ac41bdf","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150251,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150251,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150251,"SessionToken":"-1#26150251","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.564","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:18:36 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:18:36.1730349Z", "durationInMs": 0.0081},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:18:36.1730430Z", "durationInMs": 26.699},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:18:36.1997420Z", "durationInMs": 0.1964},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:18:36.1999384Z", "durationInMs": 1.8859},{"event": "Received", "startTimeUtc": "2026-04-14T20:18:36.2018243Z", "durationInMs": 0.1389},{"event": "Completed", "startTimeUtc": "2026-04-14T20:18:36.2019632Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:18:36.1886616Z","lastSend":"2026-04-14T20:18:36.1886616Z","lastReceive":"2026-04-14T20:18:36.1886616Z"},"requestSizeInBytes":1637,"requestBodySizeInBytes":1183,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:18:36.1574715Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"bae3e62a-43b2-466d-92e5-b96545e84b62","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:18:36.1574715Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"aed2369e-aec8-4ee1-bfa8-a5a6e8362928","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:18:36.1574715Z","DurationInMs":15.563400000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"c201ca2a-836f-4fa0-9884-a7086ac41bdf","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1875058,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c201ca2a-836f-4fa0-9884-a7086ac41bdf","ResponseTimeUtc":"2026-04-14T20:18:36.2042880Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c201ca2a-836f-4fa0-9884-a7086ac41bdf, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150251","BELatencyInMs":"0.564"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 20:19:33 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 96f3209f-211f-4a8b-8634-52bc780ed3e9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 96f3209f-211f-4a8b-8634-52bc780ed3e9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 96f3209f-211f-4a8b-8634-52bc780ed3e9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 96f3209f-211f-4a8b-8634-52bc780ed3e9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:19:33.161Z","duration in milliseconds":51.1193,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:19:33.0525308Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93804,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.055900000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":12.487,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.005},{"name":"Read Collection","duration in milliseconds":12.3093,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:19:33.1619711Z","DurationInMs":15.658000000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"281d1f61-3e42-4f80-800b-eb59c636dd9a","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":38.3842,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":38.344,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:18:35.0324504Z","cpu":5.382,"memory":5334580.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0995,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:18:45.0347795Z","cpu":8.047,"memory":5321948.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0923,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:18:55.0485807Z","cpu":3.510,"memory":5320544.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.09,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:19:05.0538361Z","cpu":3.438,"memory":5315796.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.096,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:19:15.0712639Z","cpu":3.276,"memory":5313316.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0729,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:19:25.0822107Z","cpu":1.092,"memory":5311096.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1013,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":38.336400000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":38.3325,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":38.3101,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":38.3076,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":37.254400000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:19:33.1776291Z","EndTimeUTC":"2026-04-14T20:19:33.1932247Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:19:33.2087781Z","DurationInMs":15.5534,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"96f3209f-211f-4a8b-8634-52bc780ed3e9","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150260,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150260,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150260,"SessionToken":"-1#26150260","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.758","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:19:33 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:19:33.1932247Z", "durationInMs": 0.0077},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:19:33.1932324Z", "durationInMs": 18.634},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:19:33.2118664Z", "durationInMs": 0.2338},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:19:33.2121002Z", "durationInMs": 2.4269},{"event": "Received", "startTimeUtc": "2026-04-14T20:19:33.2145271Z", "durationInMs": 0.1635},{"event": "Completed", "startTimeUtc": "2026-04-14T20:19:33.2146906Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:19:33.2087781Z","lastSend":"2026-04-14T20:19:33.2087781Z","lastReceive":"2026-04-14T20:19:33.2087781Z"},"requestSizeInBytes":1637,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:19:33.1776291Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"f2dade7d-d366-4668-87df-53fd7445a05a","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:19:33.1776291Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"37792971-c65f-4b29-9800-78b60d8a8fb1","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:19:33.1776291Z","DurationInMs":15.595600000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"96f3209f-211f-4a8b-8634-52bc780ed3e9","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1562473,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"96f3209f-211f-4a8b-8634-52bc780ed3e9","ResponseTimeUtc":"2026-04-14T20:19:33.2087781Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 96f3209f-211f-4a8b-8634-52bc780ed3e9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150260","BELatencyInMs":"0.758"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001}]}
2026-04-14 20:19:46 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c6870c30-ba61-4c01-b660-b8d52cf04868; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c6870c30-ba61-4c01-b660-b8d52cf04868, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c6870c30-ba61-4c01-b660-b8d52cf04868; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c6870c30-ba61-4c01-b660-b8d52cf04868, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:19:46.147Z","duration in milliseconds":42.6995,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:19:46.0379939Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93809,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0443},{"name":"Get PkValue From Stream","duration in milliseconds":6.125,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":5.9799000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:19:46.1474265Z","DurationInMs":15.566400000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"e55441bf-1134-4c9a-b04b-984f5862346b","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.3507,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.3205,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:18:55.0485807Z","cpu":3.510,"memory":5320544.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.09,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:19:05.0538361Z","cpu":3.438,"memory":5315796.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.096,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:19:15.0712639Z","cpu":3.276,"memory":5313316.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0729,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:19:25.0822107Z","cpu":1.092,"memory":5311096.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1013,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:19:35.0897218Z","cpu":4.297,"memory":5309312.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1306,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:19:45.0978599Z","cpu":3.432,"memory":5300196.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1191,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.313900000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.3102,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.2884,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.2856,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.408500000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:19:46.1629929Z","EndTimeUTC":"2026-04-14T20:19:46.1786262Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:19:46.1942459Z","DurationInMs":15.6197,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c6870c30-ba61-4c01-b660-b8d52cf04868","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150261,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150261,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150261,"SessionToken":"-1#26150261","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.784","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:19:46 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:19:46.1786262Z", "durationInMs": 0.0079},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:19:46.1786341Z", "durationInMs": 18.2694},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:19:46.1969035Z", "durationInMs": 0.2412},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:19:46.1971447Z", "durationInMs": 1.9074},{"event": "Received", "startTimeUtc": "2026-04-14T20:19:46.1990521Z", "durationInMs": 0.1468},{"event": "Completed", "startTimeUtc": "2026-04-14T20:19:46.1991989Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:19:46.1942459Z","lastSend":"2026-04-14T20:19:46.1942459Z","lastReceive":"2026-04-14T20:19:46.1942459Z"},"requestSizeInBytes":1637,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:19:46.1629929Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"9218d688-234d-422f-909a-df8c51629ab5","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:19:46.1629929Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"fced01a2-0b53-471f-92d7-356e26560cdd","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:19:46.1629929Z","DurationInMs":15.6333,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"c6870c30-ba61-4c01-b660-b8d52cf04868","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.156252,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c6870c30-ba61-4c01-b660-b8d52cf04868","ResponseTimeUtc":"2026-04-14T20:19:46.1942459Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c6870c30-ba61-4c01-b660-b8d52cf04868, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150261","BELatencyInMs":"0.784"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001}]}
2026-04-14 20:20:14 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d4353c03-9359-4feb-8d79-2204b420f4e8; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d4353c03-9359-4feb-8d79-2204b420f4e8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d4353c03-9359-4feb-8d79-2204b420f4e8; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d4353c03-9359-4feb-8d79-2204b420f4e8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:20:14.110Z","duration in milliseconds":42.8735,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:20:14.0006082Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93820,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.046},{"name":"Get PkValue From Stream","duration in milliseconds":6.2810000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.1388000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:20:14.1100565Z","DurationInMs":15.552100000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"63016cb7-a5f7-4822-99e1-881dad9016b8","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.377,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.3474,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:19:15.0712639Z","cpu":3.276,"memory":5313316.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0729,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:19:25.0822107Z","cpu":1.092,"memory":5311096.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1013,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:19:35.0897218Z","cpu":4.297,"memory":5309312.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1306,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:19:45.0978599Z","cpu":3.432,"memory":5300196.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1191,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:19:55.1058033Z","cpu":2.656,"memory":5298816.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0871,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:20:05.1185222Z","cpu":5.928,"memory":5302780.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1061,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.34,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.336800000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.316500000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.3144,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.368700000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:20:14.1412611Z","EndTimeUTC":"2026-04-14T20:20:14.1412611Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:20:14.1568605Z","DurationInMs":15.599400000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"d4353c03-9359-4feb-8d79-2204b420f4e8","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150265,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150265,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150265,"SessionToken":"-1#26150265","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.622","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:20:14 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:20:14.1412611Z", "durationInMs": 0.0081},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:20:14.1412692Z", "durationInMs": 16.8643},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:20:14.1581335Z", "durationInMs": 0.2075},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:20:14.1583410Z", "durationInMs": 1.8127},{"event": "Received", "startTimeUtc": "2026-04-14T20:20:14.1601537Z", "durationInMs": 0.1397},{"event": "Completed", "startTimeUtc": "2026-04-14T20:20:14.1602934Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:20:14.1568605Z","lastSend":"2026-04-14T20:20:14.1568605Z","lastReceive":"2026-04-14T20:20:14.1568605Z"},"requestSizeInBytes":1633,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:20:14.1256086Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"1e4b4252-3f3b-47ff-84ab-c0eb35cebd03","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:20:14.1256086Z","DurationInMs":15.652500000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"c2c117ce-a274-41ba-a51d-11030afc7de7","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:20:14.1412611Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"d4353c03-9359-4feb-8d79-2204b420f4e8","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.15625229999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"d4353c03-9359-4feb-8d79-2204b420f4e8","ResponseTimeUtc":"2026-04-14T20:20:14.1568605Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: d4353c03-9359-4feb-8d79-2204b420f4e8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150265","BELatencyInMs":"0.622"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 20:27:05 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 599a901f-2177-4b44-8114-f6fb1c983045; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 599a901f-2177-4b44-8114-f6fb1c983045, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 599a901f-2177-4b44-8114-f6fb1c983045; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 599a901f-2177-4b44-8114-f6fb1c983045, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:27:05.929Z","duration in milliseconds":42.7546,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:27:05.7415453Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":93944,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.044000000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":5.6944,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0053},{"name":"Read Collection","duration in milliseconds":5.545,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:27:05.9291262Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"102c2b20-c809-48a6-a946-a4965f2cbbe2","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.847100000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.816,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:26:15.4001622Z","cpu":4.453,"memory":5247656.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0728,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:26:25.4150590Z","cpu":15.679,"memory":5228344.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1044,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:26:35.4260763Z","cpu":3.666,"memory":5225140.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0954,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:26:45.4376759Z","cpu":4.290,"memory":5221976.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1056,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:26:55.4477025Z","cpu":3.047,"memory":5205120.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1067,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:27:05.4603108Z","cpu":2.574,"memory":5206752.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1164,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.8098,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.8038,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.787,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.7849,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.847,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:27:05.9447318Z","EndTimeUTC":"2026-04-14T20:27:05.9447318Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:27:05.9603710Z","DurationInMs":15.6392,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"599a901f-2177-4b44-8114-f6fb1c983045","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150300,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150300,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150300,"SessionToken":"-1#26150300","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.632","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:27:05 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:27:05.9447318Z", "durationInMs": 0.0096},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:27:05.9447414Z", "durationInMs": 20.2765},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:27:05.9650179Z", "durationInMs": 0.2118},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:27:05.9652297Z", "durationInMs": 1.9177},{"event": "Received", "startTimeUtc": "2026-04-14T20:27:05.9671474Z", "durationInMs": 0.1475},{"event": "Completed", "startTimeUtc": "2026-04-14T20:27:05.9672949Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:27:05.9603710Z","lastSend":"2026-04-14T20:27:05.9603710Z","lastReceive":"2026-04-14T20:27:05.9603710Z"},"requestSizeInBytes":1624,"requestBodySizeInBytes":1174,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:27:05.9291262Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"8a34aa46-f74e-488e-9666-cf8645388991","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:27:05.9291262Z","DurationInMs":15.6056,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"b2b6157a-f333-47b7-bd30-220cfdc51571","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:27:05.9447318Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"599a901f-2177-4b44-8114-f6fb1c983045","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.21882569999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"599a901f-2177-4b44-8114-f6fb1c983045","ResponseTimeUtc":"2026-04-14T20:27:05.9603710Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 599a901f-2177-4b44-8114-f6fb1c983045, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150300","BELatencyInMs":"0.632"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0015}]}
2026-04-14 20:31:39 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: a5a388aa-66d7-41b0-b023-e88b82c79ccd; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: a5a388aa-66d7-41b0-b023-e88b82c79ccd, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: a5a388aa-66d7-41b0-b023-e88b82c79ccd; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: a5a388aa-66d7-41b0-b023-e88b82c79ccd, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:31:39.476Z","duration in milliseconds":43.8624,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:31:39.4141651Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94005,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0575},{"name":"Get PkValue From Stream","duration in milliseconds":7.9851,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0024000000000000002},{"name":"Read Collection","duration in milliseconds":7.8149000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:31:39.4922935Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"2bd8fbf9-8e3c-479c-bb96-434be058ca1e","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.6351,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.5966,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:30:45.6361295Z","cpu":2.266,"memory":5164408.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2747,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:30:55.6430358Z","cpu":2.652,"memory":5164520.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0851,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:31:05.6468474Z","cpu":3.984,"memory":5157076.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0489,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:31:15.6598195Z","cpu":2.652,"memory":5192092.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1146,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:31:25.6705047Z","cpu":1.406,"memory":5188624.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0941,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:31:35.6744761Z","cpu":2.496,"memory":5186064.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1227,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.5884,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.5845,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.5657,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.5632,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.5745,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:31:39.5079759Z","EndTimeUTC":"2026-04-14T20:31:39.5079759Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:31:39.5235452Z","DurationInMs":15.5693,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"a5a388aa-66d7-41b0-b023-e88b82c79ccd","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150323,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150323,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150323,"SessionToken":"-1#26150323","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.663","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:31:39 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:31:39.5079759Z", "durationInMs": 0.0086},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:31:39.5079845Z", "durationInMs": 18.2543},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:31:39.5262388Z", "durationInMs": 0.2182},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:31:39.5264570Z", "durationInMs": 1.9937},{"event": "Received", "startTimeUtc": "2026-04-14T20:31:39.5284507Z", "durationInMs": 0.1424},{"event": "Completed", "startTimeUtc": "2026-04-14T20:31:39.5285931Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:31:39.5235452Z","lastSend":"2026-04-14T20:31:39.5235452Z","lastReceive":"2026-04-14T20:31:39.5235452Z"},"requestSizeInBytes":1625,"requestBodySizeInBytes":1175,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:31:39.4922935Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"aeb634e3-b41f-4a16-ae79-22c3165cb851","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:31:39.4922935Z","DurationInMs":15.682400000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a7d030c9-3bd3-423a-96f0-86b3750b460f","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:31:39.5079759Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"a5a388aa-66d7-41b0-b023-e88b82c79ccd","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1093801,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"a5a388aa-66d7-41b0-b023-e88b82c79ccd","ResponseTimeUtc":"2026-04-14T20:31:39.5235452Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: a5a388aa-66d7-41b0-b023-e88b82c79ccd, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150323","BELatencyInMs":"0.663"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 20:56:37 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b2299d4f-65a5-41af-9eac-ab5125e40030; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b2299d4f-65a5-41af-9eac-ab5125e40030, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: b2299d4f-65a5-41af-9eac-ab5125e40030; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: b2299d4f-65a5-41af-9eac-ab5125e40030, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:56:37.803Z","duration in milliseconds":52.820100000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:56:37.7255359Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94283,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0536},{"name":"Get PkValue From Stream","duration in milliseconds":6.7607,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0023},{"name":"Read Collection","duration in milliseconds":6.5949,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:56:37.8037261Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"9d98c683-8ba3-4922-822e-d00e1bf6ae2c","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":45.8172,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":45.7792,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:55:46.9424527Z","cpu":2.813,"memory":5267944.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:55:56.9492032Z","cpu":1.797,"memory":5266668.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3554,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:56:06.9531079Z","cpu":5.850,"memory":5232908.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.099,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:56:16.9582734Z","cpu":0.938,"memory":5237352.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1025,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:56:26.9619181Z","cpu":3.516,"memory":5239716.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0553,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:56:36.9755988Z","cpu":2.808,"memory":5231780.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1042,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":45.7723,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":45.7686,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":45.7451,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":45.7424,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":44.7805,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:56:37.8193740Z","EndTimeUTC":"2026-04-14T20:56:37.8193740Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:56:37.8505866Z","DurationInMs":31.212600000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"b2299d4f-65a5-41af-9eac-ab5125e40030","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150612,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150612,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150612,"SessionToken":"-1#26150612","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.868","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:56:37 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:56:37.8193740Z", "durationInMs": 0.0095},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:56:37.8193835Z", "durationInMs": 27.9142},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:56:37.8472977Z", "durationInMs": 0.1983},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:56:37.8474960Z", "durationInMs": 2.0362},{"event": "Received", "startTimeUtc": "2026-04-14T20:56:37.8495322Z", "durationInMs": 0.142},{"event": "Completed", "startTimeUtc": "2026-04-14T20:56:37.8496742Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:56:37.8505866Z","lastSend":"2026-04-14T20:56:37.8505866Z","lastReceive":"2026-04-14T20:56:37.8505866Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:56:37.8037261Z","DurationInMs":15.6479,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a6d2d2e9-1130-4a70-94cb-c11498cd0419","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:56:37.8193740Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"3a1346a1-d37b-4b28-aff2-2eafa70beb2d","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:56:37.8193740Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"b2299d4f-65a5-41af-9eac-ab5125e40030","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1250507,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"b2299d4f-65a5-41af-9eac-ab5125e40030","ResponseTimeUtc":"2026-04-14T20:56:37.8505866Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: b2299d4f-65a5-41af-9eac-ab5125e40030, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150612","BELatencyInMs":"0.868"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 20:58:23 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c21cd25d-e5ba-4bef-9a7d-ad07b09d22a9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c21cd25d-e5ba-4bef-9a7d-ad07b09d22a9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c21cd25d-e5ba-4bef-9a7d-ad07b09d22a9; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c21cd25d-e5ba-4bef-9a7d-ad07b09d22a9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:58:23.151Z","duration in milliseconds":47.741400000000006,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:58:23.0264796Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94299,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.056},{"name":"Get PkValue From Stream","duration in milliseconds":6.7744,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002},{"name":"Read Collection","duration in milliseconds":6.5368,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:58:23.1515733Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"6b652ede-21fa-488b-8e23-dfe849eff2c1","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":40.713100000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":40.6665,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:57:27.0176261Z","cpu":2.496,"memory":5226332.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0804,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:57:37.0221715Z","cpu":1.484,"memory":5225940.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0963,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:57:47.0260334Z","cpu":3.281,"memory":5236600.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0978,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:57:57.0364872Z","cpu":3.042,"memory":5241040.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1157,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:58:07.0492287Z","cpu":2.964,"memory":5289724.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1296,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:58:17.0607059Z","cpu":2.266,"memory":5287936.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0945,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":40.6573,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":40.653200000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":40.628,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":40.623000000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":39.596000000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:58:23.1671105Z","EndTimeUTC":"2026-04-14T20:58:23.1671105Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:58:23.1983603Z","DurationInMs":31.2498,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c21cd25d-e5ba-4bef-9a7d-ad07b09d22a9","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150617,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150617,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150617,"SessionToken":"-1#26150617","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.607","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:58:23 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:58:23.1671105Z", "durationInMs": 0.0087},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:58:23.1671192Z", "durationInMs": 21.413},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:58:23.1885322Z", "durationInMs": 0.2236},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:58:23.1887558Z", "durationInMs": 1.8131},{"event": "Received", "startTimeUtc": "2026-04-14T20:58:23.1905689Z", "durationInMs": 0.1508},{"event": "Completed", "startTimeUtc": "2026-04-14T20:58:23.1907197Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:58:23.1827364Z","lastSend":"2026-04-14T20:58:23.1827364Z","lastReceive":"2026-04-14T20:58:23.1983603Z"},"requestSizeInBytes":1642,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:58:23.1515733Z","DurationInMs":15.5372,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"d4a8bae7-b4ed-4e0f-9b4c-8e360507ba08","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:58:23.1671105Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"7337c4f1-2d2c-49d7-a4d2-be843ddd844d","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:58:23.1671105Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"c21cd25d-e5ba-4bef-9a7d-ad07b09d22a9","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1718807,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c21cd25d-e5ba-4bef-9a7d-ad07b09d22a9","ResponseTimeUtc":"2026-04-14T20:58:23.1983603Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c21cd25d-e5ba-4bef-9a7d-ad07b09d22a9, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150617","BELatencyInMs":"0.607"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.003}]}
2026-04-14 20:58:41 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 1b44674b-c7f9-4bac-abb6-30064ab420ef; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 1b44674b-c7f9-4bac-abb6-30064ab420ef, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 1b44674b-c7f9-4bac-abb6-30064ab420ef; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 1b44674b-c7f9-4bac-abb6-30064ab420ef, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:58:41.804Z","duration in milliseconds":41.542100000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:58:41.6637511Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94304,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.046},{"name":"Get PkValue From Stream","duration in milliseconds":7.0826,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":6.9388000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:58:41.8043713Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"5d82e870-cbb0-413b-adbd-7ad9e34087a3","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":34.229800000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":34.1976,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:57:47.0260334Z","cpu":3.281,"memory":5236600.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0978,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:57:57.0364872Z","cpu":3.042,"memory":5241040.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1157,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:58:07.0492287Z","cpu":2.964,"memory":5289724.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1296,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:58:17.0607059Z","cpu":2.266,"memory":5287936.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0945,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:58:27.0671803Z","cpu":2.808,"memory":5296944.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0863,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:58:37.0772541Z","cpu":1.719,"memory":5294908.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1322,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":34.1878,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":34.1837,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":34.1642,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":34.1619,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":33.2787,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:58:41.8200408Z","EndTimeUTC":"2026-04-14T20:58:41.8200408Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:58:41.8512521Z","DurationInMs":31.2113,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"1b44674b-c7f9-4bac-abb6-30064ab420ef","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150618,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150618,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150618,"SessionToken":"-1#26150618","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.602","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:58:41 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:58:41.8200408Z", "durationInMs": 0.0087},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:58:41.8200495Z", "durationInMs": 15.4036},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:58:41.8354531Z", "durationInMs": 0.2228},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:58:41.8356759Z", "durationInMs": 1.996},{"event": "Received", "startTimeUtc": "2026-04-14T20:58:41.8376719Z", "durationInMs": 0.1502},{"event": "Completed", "startTimeUtc": "2026-04-14T20:58:41.8378221Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:58:41.8356355Z","lastSend":"2026-04-14T20:58:41.8356355Z","lastReceive":"2026-04-14T20:58:41.8356355Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:58:41.8043713Z","DurationInMs":15.669500000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"7daeb320-c3b7-4092-a711-8fa0204c1441","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:58:41.8200408Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"1d91a3fe-c37d-4ff2-b7d0-1e4e1682c19c","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:58:41.8200408Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"1b44674b-c7f9-4bac-abb6-30064ab420ef","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.187501,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"1b44674b-c7f9-4bac-abb6-30064ab420ef","ResponseTimeUtc":"2026-04-14T20:58:41.8512521Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 1b44674b-c7f9-4bac-abb6-30064ab420ef, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150618","BELatencyInMs":"0.602"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 20:58:59 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e0006969-3e41-46be-bc38-f7aedd6f49d8; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e0006969-3e41-46be-bc38-f7aedd6f49d8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e0006969-3e41-46be-bc38-f7aedd6f49d8; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e0006969-3e41-46be-bc38-f7aedd6f49d8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:58:59.546Z","duration in milliseconds":42.8654,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:58:59.4213537Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94308,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.049},{"name":"Get PkValue From Stream","duration in milliseconds":6.1913,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.029,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:58:59.5464195Z","DurationInMs":15.5534,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"d45a4952-7c5d-4a63-83dd-0d8474d175b9","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.373000000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.3354,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:58:07.0492287Z","cpu":2.964,"memory":5289724.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1296,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:58:17.0607059Z","cpu":2.266,"memory":5287936.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0945,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:58:27.0671803Z","cpu":2.808,"memory":5296944.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0863,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:58:37.0772541Z","cpu":1.719,"memory":5294908.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1322,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:58:47.0808605Z","cpu":4.680,"memory":5291328.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0842,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:58:57.0945629Z","cpu":4.056,"memory":5264380.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0937,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.3224,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.3179,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.2896,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.286500000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.3086,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:58:59.5619729Z","EndTimeUTC":"2026-04-14T20:58:59.5776014Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:58:59.5932811Z","DurationInMs":15.6797,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"e0006969-3e41-46be-bc38-f7aedd6f49d8","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150618,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150618,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150618,"SessionToken":"-1#26150618","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.744","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:58:59 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:58:59.5776014Z", "durationInMs": 0.0072},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:58:59.5776086Z", "durationInMs": 15.3785},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:58:59.5929871Z", "durationInMs": 0.2229},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:58:59.5932100Z", "durationInMs": 2.0668},{"event": "Received", "startTimeUtc": "2026-04-14T20:58:59.5952768Z", "durationInMs": 0.1576},{"event": "Completed", "startTimeUtc": "2026-04-14T20:58:59.5954344Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:58:59.5932811Z","lastSend":"2026-04-14T20:58:59.5932811Z","lastReceive":"2026-04-14T20:58:59.5932811Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:58:59.5619729Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a3ffe641-6391-49d5-9407-26d8a220178b","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:58:59.5619729Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"e1f335c8-3faf-447a-a22d-29efa012fd52","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:58:59.5619729Z","DurationInMs":15.6285,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"e0006969-3e41-46be-bc38-f7aedd6f49d8","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.17192739999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"e0006969-3e41-46be-bc38-f7aedd6f49d8","ResponseTimeUtc":"2026-04-14T20:58:59.5932811Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: e0006969-3e41-46be-bc38-f7aedd6f49d8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150618","BELatencyInMs":"0.744"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0024000000000000002}]}
2026-04-14 20:59:22 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e2f5b4d5-a902-4979-ab83-22eafdc28bc5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e2f5b4d5-a902-4979-ab83-22eafdc28bc5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e2f5b4d5-a902-4979-ab83-22eafdc28bc5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e2f5b4d5-a902-4979-ab83-22eafdc28bc5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:59:22.376Z","duration in milliseconds":44.6888,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:59:22.2512212Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94314,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0448},{"name":"Get PkValue From Stream","duration in milliseconds":6.8553000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0019},{"name":"Read Collection","duration in milliseconds":6.6892000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:59:22.3762235Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"2951ef3d-d5ec-4902-bdec-68bec6db3349","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.6087,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.5711,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:58:27.0671803Z","cpu":2.808,"memory":5296944.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0863,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:58:37.0772541Z","cpu":1.719,"memory":5294908.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1322,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:58:47.0808605Z","cpu":4.680,"memory":5291328.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0842,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:58:57.0945629Z","cpu":4.056,"memory":5264380.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0937,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:59:07.1060506Z","cpu":7.813,"memory":5261364.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0916,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:59:17.1130236Z","cpu":7.956,"memory":5259012.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1198,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.5621,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.5569,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.534800000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.5321,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.572500000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:59:22.3918472Z","EndTimeUTC":"2026-04-14T20:59:22.3918472Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:59:22.4231004Z","DurationInMs":31.253200000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"e2f5b4d5-a902-4979-ab83-22eafdc28bc5","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150624,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150624,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150624,"SessionToken":"-1#26150624","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.574","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:59:22 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:59:22.3918472Z", "durationInMs": 0.008},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:59:22.3918552Z", "durationInMs": 18.7783},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:59:22.4106335Z", "durationInMs": 0.2513},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:59:22.4108848Z", "durationInMs": 1.8139},{"event": "Received", "startTimeUtc": "2026-04-14T20:59:22.4126987Z", "durationInMs": 0.1346},{"event": "Completed", "startTimeUtc": "2026-04-14T20:59:22.4128333Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:59:22.4074750Z","lastSend":"2026-04-14T20:59:22.4074750Z","lastReceive":"2026-04-14T20:59:22.4074750Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:59:22.3762235Z","DurationInMs":15.623700000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"470fef32-5a92-4038-81c0-9ba3e7b17274","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:59:22.3918472Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"7e2e1910-e99f-48b1-932c-030cf17c5107","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:59:22.3918472Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"e2f5b4d5-a902-4979-ab83-22eafdc28bc5","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.17187919999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"e2f5b4d5-a902-4979-ab83-22eafdc28bc5","ResponseTimeUtc":"2026-04-14T20:59:22.4231004Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: e2f5b4d5-a902-4979-ab83-22eafdc28bc5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150624","BELatencyInMs":"0.574"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0015}]}
2026-04-14 20:59:39 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 7479b2ef-eccd-44ae-b451-6b4446fd7b52; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 7479b2ef-eccd-44ae-b451-6b4446fd7b52, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 7479b2ef-eccd-44ae-b451-6b4446fd7b52; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 7479b2ef-eccd-44ae-b451-6b4446fd7b52, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:59:39.622Z","duration in milliseconds":49.1103,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:59:39.4974609Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94319,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.053000000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":7.4708000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0016},{"name":"Read Collection","duration in milliseconds":7.2890000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:59:39.6224623Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"ad8ab4ab-182a-4ee6-8944-d197f9e8076a","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":41.386,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":41.3449,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:58:47.0808605Z","cpu":4.680,"memory":5291328.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0842,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:58:57.0945629Z","cpu":4.056,"memory":5264380.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0937,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:59:07.1060506Z","cpu":7.813,"memory":5261364.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0916,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:59:17.1130236Z","cpu":7.956,"memory":5259012.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1198,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:59:27.1190891Z","cpu":8.672,"memory":5259764.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0865,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:59:37.1341424Z","cpu":6.708,"memory":5259816.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0984,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":41.335100000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":41.3311,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":41.310300000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":41.307300000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":40.4132,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:59:39.6380870Z","EndTimeUTC":"2026-04-14T20:59:39.6380870Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:59:39.6693369Z","DurationInMs":31.2499,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"7479b2ef-eccd-44ae-b451-6b4446fd7b52","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150625,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150625,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150625,"SessionToken":"-1#26150625","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.677","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:59:39 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:59:39.6380870Z", "durationInMs": 0.0078},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:59:39.6380948Z", "durationInMs": 19.8185},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:59:39.6579133Z", "durationInMs": 0.2102},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:59:39.6581235Z", "durationInMs": 3.7479},{"event": "Received", "startTimeUtc": "2026-04-14T20:59:39.6618714Z", "durationInMs": 0.1666},{"event": "Completed", "startTimeUtc": "2026-04-14T20:59:39.6620380Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:59:39.6693369Z","lastSend":"2026-04-14T20:59:39.6693369Z","lastReceive":"2026-04-14T20:59:39.6693369Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:59:39.6224623Z","DurationInMs":15.6247,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"9e0f6fe1-df64-4f9a-ba3d-e641c6773af3","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:59:39.6380870Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"4629ec7a-8841-419b-b03a-773aa504848e","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:59:39.6380870Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"7479b2ef-eccd-44ae-b451-6b4446fd7b52","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.171876,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"7479b2ef-eccd-44ae-b451-6b4446fd7b52","ResponseTimeUtc":"2026-04-14T20:59:39.6693369Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 7479b2ef-eccd-44ae-b451-6b4446fd7b52, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150625","BELatencyInMs":"0.677"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0032}]}
2026-04-14 20:59:56 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 937323d9-fae5-4b20-8875-b34908b8cb70; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 937323d9-fae5-4b20-8875-b34908b8cb70, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 937323d9-fae5-4b20-8875-b34908b8cb70; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 937323d9-fae5-4b20-8875-b34908b8cb70, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T20:59:56.739Z","duration in milliseconds":45.832100000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T20:59:56.5829578Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94322,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0512},{"name":"Get PkValue From Stream","duration in milliseconds":8.4339000000000013,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014},{"name":"Read Collection","duration in milliseconds":8.2695,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:59:56.7392775Z","DurationInMs":15.630500000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"b535ffbb-5b30-4626-ad74-6a2c5d143309","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.1669,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.1267,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:58:57.0945629Z","cpu":4.056,"memory":5264380.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0937,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:59:07.1060506Z","cpu":7.813,"memory":5261364.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0916,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:59:17.1130236Z","cpu":7.956,"memory":5259012.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1198,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:59:27.1190891Z","cpu":8.672,"memory":5259764.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0865,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:59:37.1341424Z","cpu":6.708,"memory":5259816.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0984,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:59:47.1382841Z","cpu":8.125,"memory":5260392.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0845,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.1194,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.1154,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.0947,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.090900000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.185,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T20:59:56.7549080Z","EndTimeUTC":"2026-04-14T20:59:56.7704661Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T20:59:56.7860887Z","DurationInMs":15.6226,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"937323d9-fae5-4b20-8875-b34908b8cb70","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150626,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150626,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150626,"SessionToken":"-1#26150626","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.609","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 8:59:56 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T20:59:56.7704661Z", "durationInMs": 0.0086},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T20:59:56.7704747Z", "durationInMs": 17.787},{"event": "Pipelined", "startTimeUtc": "2026-04-14T20:59:56.7882617Z", "durationInMs": 0.1987},{"event": "Transit Time", "startTimeUtc": "2026-04-14T20:59:56.7884604Z", "durationInMs": 2.5762},{"event": "Received", "startTimeUtc": "2026-04-14T20:59:56.7910366Z", "durationInMs": 0.1412},{"event": "Completed", "startTimeUtc": "2026-04-14T20:59:56.7911778Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T20:59:56.7860887Z","lastSend":"2026-04-14T20:59:56.7860887Z","lastReceive":"2026-04-14T20:59:56.7860887Z"},"requestSizeInBytes":1630,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T20:59:56.7549080Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a7425f45-c917-4cdd-abbb-75c8f2bc38f5","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T20:59:56.7549080Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"66b8d38e-312f-4c38-8637-4a56e5ccee8b","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T20:59:56.7549080Z","DurationInMs":15.558100000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"937323d9-fae5-4b20-8875-b34908b8cb70","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2031309,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"937323d9-fae5-4b20-8875-b34908b8cb70","ResponseTimeUtc":"2026-04-14T20:59:56.7860887Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 937323d9-fae5-4b20-8875-b34908b8cb70, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150626","BELatencyInMs":"0.609"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 21:00:16 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: f0d4f4b9-1be9-4547-a8f8-5466166ef226; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: f0d4f4b9-1be9-4547-a8f8-5466166ef226, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: f0d4f4b9-1be9-4547-a8f8-5466166ef226; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: f0d4f4b9-1be9-4547-a8f8-5466166ef226, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:00:16.332Z","duration in milliseconds":56.409200000000006,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:00:16.1763654Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94327,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.048100000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":6.6282000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0019},{"name":"Read Collection","duration in milliseconds":6.4743,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:00:16.3326986Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"3ea12fd6-7471-49ce-8381-7094e6fd8bc9","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":49.5512,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":49.515,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T20:59:17.1130236Z","cpu":7.956,"memory":5259012.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1198,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:59:27.1190891Z","cpu":8.672,"memory":5259764.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0865,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:59:37.1341424Z","cpu":6.708,"memory":5259816.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0984,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:59:47.1382841Z","cpu":8.125,"memory":5260392.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0845,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T20:59:57.1454843Z","cpu":7.176,"memory":5258512.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2679,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:00:07.1616199Z","cpu":16.303,"memory":5251064.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1029,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":49.5067,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":49.5024,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":49.4812,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":49.478100000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":48.6235,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:00:16.3639555Z","EndTimeUTC":"2026-04-14T21:00:16.3639555Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:00:16.3951115Z","DurationInMs":31.156000000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"f0d4f4b9-1be9-4547-a8f8-5466166ef226","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150628,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150628,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150628,"SessionToken":"-1#26150628","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.607","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:00:16 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:00:16.3639555Z", "durationInMs": 0.0088},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:00:16.3639643Z", "durationInMs": 18.0757},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:00:16.3820400Z", "durationInMs": 0.2282},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:00:16.3822682Z", "durationInMs": 1.8041},{"event": "Received", "startTimeUtc": "2026-04-14T21:00:16.3840723Z", "durationInMs": 0.1486},{"event": "Completed", "startTimeUtc": "2026-04-14T21:00:16.3842209Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:00:16.3794870Z","lastSend":"2026-04-14T21:00:16.3794870Z","lastReceive":"2026-04-14T21:00:16.3794870Z"},"requestSizeInBytes":1630,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:00:16.3326986Z","DurationInMs":15.539700000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"6bda88df-4987-4ab8-af41-172bc2ebb5ac","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:00:16.3482383Z","DurationInMs":15.7172,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"c05a82c1-5523-4848-b1fc-3078786d5bdd","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:00:16.3639555Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"f0d4f4b9-1be9-4547-a8f8-5466166ef226","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2187461,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"f0d4f4b9-1be9-4547-a8f8-5466166ef226","ResponseTimeUtc":"2026-04-14T21:00:16.3951115Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: f0d4f4b9-1be9-4547-a8f8-5466166ef226, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150628","BELatencyInMs":"0.607"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0033}]}
2026-04-14 21:06:12 | Log4Net | Unhandled exception in DiscardArchivalMaterials: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 928a81fc-5d88-43f3-a8f0-3eb937c61d9a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 928a81fc-5d88-43f3-a8f0-3eb937c61d9a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 928a81fc-5d88-43f3-a8f0-3eb937c61d9a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 928a81fc-5d88-43f3-a8f0-3eb937c61d9a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:06:12.933Z","duration in milliseconds":4.1608,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:06:11.9956889Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94375,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.053500000000000006},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":3.9471000000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":3.9016,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:05:17.4426561Z","cpu":5.156,"memory":5229972.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.107,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:05:27.4516416Z","cpu":1.560,"memory":5231448.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.118,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:05:37.4657749Z","cpu":1.794,"memory":5229304.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.098,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:05:47.4696546Z","cpu":1.406,"memory":5229540.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1221,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:05:57.4719380Z","cpu":2.031,"memory":5227824.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1003,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:06:07.4873360Z","cpu":4.992,"memory":5186884.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0937,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":3.8942,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":3.8897000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":3.8709000000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":3.8677,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":2.8409,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:06:12.9333305Z","DurationInMs":0,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"928a81fc-5d88-43f3-a8f0-3eb937c61d9a","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150639,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150639,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150639,"SessionToken":"-1#26150639","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.606","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:06:12 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:06:12.9333305Z", "durationInMs": 0.0038},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:06:12.9333343Z", "durationInMs": 0.0047},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:06:12.9333390Z", "durationInMs": 0.2446},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:06:12.9335836Z", "durationInMs": 1.7908},{"event": "Received", "startTimeUtc": "2026-04-14T21:06:12.9353744Z", "durationInMs": 0.0707},{"event": "Completed", "startTimeUtc": "2026-04-14T21:06:12.9354451Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:06:12.9019565Z","lastSend":"2026-04-14T21:06:12.9019565Z","lastReceive":"2026-04-14T21:06:12.9333305Z"},"requestSizeInBytes":1587,"requestBodySizeInBytes":1139,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.9376416,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"928a81fc-5d88-43f3-a8f0-3eb937c61d9a","ResponseTimeUtc":"2026-04-14T21:06:12.9333305Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 928a81fc-5d88-43f3-a8f0-3eb937c61d9a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150639","BELatencyInMs":"0.606"}}}]}]}]}]}]}]}]}
2026-04-14 21:25:01 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 8b5c3590-e50a-4167-b044-8005a307b3dd; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 8b5c3590-e50a-4167-b044-8005a307b3dd, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 8b5c3590-e50a-4167-b044-8005a307b3dd; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 8b5c3590-e50a-4167-b044-8005a307b3dd, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:25:01.518Z","duration in milliseconds":44.646300000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:25:00.6587549Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94640,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0495},{"name":"Get PkValue From Stream","duration in milliseconds":6.6541000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.4958,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:25:01.5186932Z","DurationInMs":15.6286,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"062a7ef5-2d1b-47e2-8b88-925ce672b454","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.712900000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.6773,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:24:08.3600160Z","cpu":3.198,"memory":5337756.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1292,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:24:18.3714579Z","cpu":3.672,"memory":5330600.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0931,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:24:28.3855644Z","cpu":0.936,"memory":5336788.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0879,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:24:38.3869876Z","cpu":0.547,"memory":5332948.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1205,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:24:48.3894413Z","cpu":2.574,"memory":5327920.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1079,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:24:58.3931594Z","cpu":2.969,"memory":5311308.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0807,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.669000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.6652,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.6432,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.6403,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.3491,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:25:01.5343218Z","EndTimeUTC":"2026-04-14T21:25:01.5499848Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:25:01.5655157Z","DurationInMs":15.5309,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"8b5c3590-e50a-4167-b044-8005a307b3dd","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150694,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150694,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150694,"SessionToken":"-1#26150694","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.719","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:25:01 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:25:01.5499848Z", "durationInMs": 0.0106},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:25:01.5499954Z", "durationInMs": 17.1389},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:25:01.5671343Z", "durationInMs": 0.1981},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:25:01.5673324Z", "durationInMs": 1.806},{"event": "Received", "startTimeUtc": "2026-04-14T21:25:01.5691384Z", "durationInMs": 0.1453},{"event": "Completed", "startTimeUtc": "2026-04-14T21:25:01.5692837Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:25:01.5655157Z","lastSend":"2026-04-14T21:25:01.5655157Z","lastReceive":"2026-04-14T21:25:01.5655157Z"},"requestSizeInBytes":1625,"requestBodySizeInBytes":1173,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:25:01.5343218Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"ffe50ccf-5db7-4519-9b3d-d7a5e8d35be8","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:25:01.5343218Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"b897a847-cbf1-4142-8143-7ccf2797ead0","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:25:01.5343218Z","DurationInMs":15.663,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"8b5c3590-e50a-4167-b044-8005a307b3dd","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.90676079999999992,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"8b5c3590-e50a-4167-b044-8005a307b3dd","ResponseTimeUtc":"2026-04-14T21:25:01.5655157Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 8b5c3590-e50a-4167-b044-8005a307b3dd, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150694","BELatencyInMs":"0.719"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 21:30:55 | Log4Net | Unhandled exception in DiscardArchivalMaterials: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 1c0926b2-ac81-4c27-b7b7-712e8bd5e5b4; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 1c0926b2-ac81-4c27-b7b7-712e8bd5e5b4, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 1c0926b2-ac81-4c27-b7b7-712e8bd5e5b4; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 1c0926b2-ac81-4c27-b7b7-712e8bd5e5b4, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:30:55.139Z","duration in milliseconds":4.7941,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:30:54.2797221Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94714,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.055600000000000004},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":4.5597,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":4.5176,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:29:58.6355119Z","cpu":1.248,"memory":5172716.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0896,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:30:08.6463544Z","cpu":9.984,"memory":5201372.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.104,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1},{"dateUtc":"2026-04-14T21:30:18.6579905Z","cpu":7.500,"memory":5153412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0624,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:30:28.6679991Z","cpu":2.262,"memory":5152736.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0957,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:30:38.6724217Z","cpu":3.125,"memory":5166176.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.114,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:30:48.6814713Z","cpu":4.446,"memory":5165820.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0557,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":4.5093000000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":4.5025,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":4.4821,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":4.4788000000000006,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":3.3737000000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:30:55.1392375Z","DurationInMs":0,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"1c0926b2-ac81-4c27-b7b7-712e8bd5e5b4","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150725,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150725,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150725,"SessionToken":"-1#26150725","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.715","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:30:55 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:30:55.1392375Z", "durationInMs": 0.0036},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:30:55.1392411Z", "durationInMs": 0.0053},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:30:55.1392464Z", "durationInMs": 0.2254},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:30:55.1394718Z", "durationInMs": 2.3503},{"event": "Received", "startTimeUtc": "2026-04-14T21:30:55.1418221Z", "durationInMs": 0.0744},{"event": "Completed", "startTimeUtc": "2026-04-14T21:30:55.1418965Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:30:55.1234805Z","lastSend":"2026-04-14T21:30:55.1234805Z","lastReceive":"2026-04-14T21:30:55.1392375Z"},"requestSizeInBytes":1590,"requestBodySizeInBytes":1140,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.85951539999999993,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"1c0926b2-ac81-4c27-b7b7-712e8bd5e5b4","ResponseTimeUtc":"2026-04-14T21:30:55.1392375Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 1c0926b2-ac81-4c27-b7b7-712e8bd5e5b4, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150725","BELatencyInMs":"0.715"}}}]}]}]}]}]}]}]}
2026-04-14 21:45:09 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 522fea76-1c74-4da0-865b-50b36407127f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 522fea76-1c74-4da0-865b-50b36407127f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 522fea76-1c74-4da0-865b-50b36407127f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 522fea76-1c74-4da0-865b-50b36407127f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:45:09.179Z","duration in milliseconds":47.3219,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:45:08.3355736Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94865,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.04},{"name":"Get PkValue From Stream","duration in milliseconds":6.0377,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0035},{"name":"Read Collection","duration in milliseconds":5.9035,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:45:09.1949645Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"8729600b-d760-485f-8aa3-1b2cdae8748d","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":41.0249,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":40.9902,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:44:09.3595812Z","cpu":5.313,"memory":5270176.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1082,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:44:19.3676712Z","cpu":3.432,"memory":5268916.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1061,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:44:29.3722211Z","cpu":3.828,"memory":5253740.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0947,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:44:39.3874975Z","cpu":5.694,"memory":5351368.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":1.6402,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:44:49.4006739Z","cpu":2.574,"memory":5344960.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1211,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:44:59.4090478Z","cpu":14.531,"memory":5248764.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1183,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":40.9802,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":40.9748,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":40.9513,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":40.9234,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":39.5805,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:45:09.2105852Z","EndTimeUTC":"2026-04-14T21:45:09.2105852Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:45:09.2262143Z","DurationInMs":15.629100000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"522fea76-1c74-4da0-865b-50b36407127f","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150786,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150786,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150786,"SessionToken":"-1#26150786","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.794","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:45:09 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:45:09.2105852Z", "durationInMs": 0.0131},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:45:09.2105983Z", "durationInMs": 21.3201},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:45:09.2319184Z", "durationInMs": 0.2273},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:45:09.2321457Z", "durationInMs": 2.0471},{"event": "Received", "startTimeUtc": "2026-04-14T21:45:09.2341928Z", "durationInMs": 0.1496},{"event": "Completed", "startTimeUtc": "2026-04-14T21:45:09.2343424Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:45:09.2262143Z","lastSend":"2026-04-14T21:45:09.2262143Z","lastReceive":"2026-04-14T21:45:09.2262143Z"},"requestSizeInBytes":1627,"requestBodySizeInBytes":1175,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:45:09.1949645Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"d45ff6c6-d2bb-44c5-991e-6f3945a47ba7","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:45:09.1949645Z","DurationInMs":15.620700000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0e64fce2-6770-4ed4-97d1-e7f66d0406b4","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:45:09.2105852Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"522fea76-1c74-4da0-865b-50b36407127f","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.8906407,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"522fea76-1c74-4da0-865b-50b36407127f","ResponseTimeUtc":"2026-04-14T21:45:09.2262143Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 522fea76-1c74-4da0-865b-50b36407127f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150786","BELatencyInMs":"0.794"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 21:50:03 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d2783b9f-df2d-4b70-8d77-a88d2070f0f6; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d2783b9f-df2d-4b70-8d77-a88d2070f0f6, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d2783b9f-df2d-4b70-8d77-a88d2070f0f6; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d2783b9f-df2d-4b70-8d77-a88d2070f0f6, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:50:03.804Z","duration in milliseconds":50.707,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:50:03.6793296Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94912,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0459},{"name":"Get PkValue From Stream","duration in milliseconds":8.6325,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":8.4816,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:50:03.8043929Z","DurationInMs":15.5744,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"217366f8-748d-404e-b03c-2db634189bc1","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":41.845,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":41.815000000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:49:09.6045769Z","cpu":4.290,"memory":5273376.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2732,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:49:19.6165416Z","cpu":0.547,"memory":5272980.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0677,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:49:29.6246767Z","cpu":2.496,"memory":5271748.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2576,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:49:39.6300968Z","cpu":2.188,"memory":5272124.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.102,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:49:49.6336025Z","cpu":0.547,"memory":5264744.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0792,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:49:59.6476698Z","cpu":2.808,"memory":5264120.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1129,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":41.8095,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":41.8044,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":41.7858,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":41.7837,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":40.8036,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:50:03.8199673Z","EndTimeUTC":"2026-04-14T21:50:03.8355886Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:50:03.8512156Z","DurationInMs":15.627,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"d2783b9f-df2d-4b70-8d77-a88d2070f0f6","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150810,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150810,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150810,"SessionToken":"-1#26150810","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"1.943","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:50:03 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:50:03.8355886Z", "durationInMs": 0.0089},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:50:03.8355975Z", "durationInMs": 23.142},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:50:03.8587395Z", "durationInMs": 0.1953},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:50:03.8589348Z", "durationInMs": 3.1984},{"event": "Received", "startTimeUtc": "2026-04-14T21:50:03.8621332Z", "durationInMs": 0.1366},{"event": "Completed", "startTimeUtc": "2026-04-14T21:50:03.8622698Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:50:03.8512156Z","lastSend":"2026-04-14T21:50:03.8512156Z","lastReceive":"2026-04-14T21:50:03.8512156Z"},"requestSizeInBytes":1635,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:50:03.8199673Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"8634f74c-a633-468c-96e3-df40c50d7943","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:50:03.8199673Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"d7cf1470-822e-40d8-ac8e-f0b005b4acdf","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:50:03.8199673Z","DurationInMs":15.621300000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"d2783b9f-df2d-4b70-8d77-a88d2070f0f6","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.17188599999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"d2783b9f-df2d-4b70-8d77-a88d2070f0f6","ResponseTimeUtc":"2026-04-14T21:50:03.8512156Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: d2783b9f-df2d-4b70-8d77-a88d2070f0f6, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150810","BELatencyInMs":"1.943"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 21:50:13 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: a69cd7fa-c753-492c-af9c-eee806420850; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: a69cd7fa-c753-492c-af9c-eee806420850, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: a69cd7fa-c753-492c-af9c-eee806420850; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: a69cd7fa-c753-492c-af9c-eee806420850, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:50:13.948Z","duration in milliseconds":42.915600000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:50:13.6831368Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94914,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0489},{"name":"Get PkValue From Stream","duration in milliseconds":6.8652000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001},{"name":"Read Collection","duration in milliseconds":6.7156,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:50:13.9487661Z","DurationInMs":15.732700000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"ff6269a5-5520-48d2-84a8-bedd94ab2631","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.8297,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.7977,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:49:19.6165416Z","cpu":0.547,"memory":5272980.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0677,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:49:29.6246767Z","cpu":2.496,"memory":5271748.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2576,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:49:39.6300968Z","cpu":2.188,"memory":5272124.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.102,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:49:49.6336025Z","cpu":0.547,"memory":5264744.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0792,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:49:59.6476698Z","cpu":2.808,"memory":5264120.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1129,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:50:09.6549407Z","cpu":3.900,"memory":5272540.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1002,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.7907,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.7867,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.766400000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.7637,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.908500000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:50:13.9644988Z","EndTimeUTC":"2026-04-14T21:50:13.9800159Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:50:13.9956405Z","DurationInMs":15.624600000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"a69cd7fa-c753-492c-af9c-eee806420850","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150810,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150810,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150810,"SessionToken":"-1#26150810","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.632","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:50:13 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:50:13.9800159Z", "durationInMs": 0.0082},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:50:13.9800241Z", "durationInMs": 18.4317},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:50:13.9984558Z", "durationInMs": 0.1896},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:50:13.9986454Z", "durationInMs": 1.9378},{"event": "Received", "startTimeUtc": "2026-04-14T21:50:14.0005832Z", "durationInMs": 0.1534},{"event": "Completed", "startTimeUtc": "2026-04-14T21:50:14.0007366Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:50:13.9956405Z","lastSend":"2026-04-14T21:50:13.9956405Z","lastReceive":"2026-04-14T21:50:13.9956405Z"},"requestSizeInBytes":1610,"requestBodySizeInBytes":1158,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:50:13.9644988Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"eb5a8593-7632-4ac9-9b27-5c3b7e627bf2","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:50:13.9644988Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"6cdc4525-527c-49da-82fd-1013721e5a98","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:50:13.9644988Z","DurationInMs":15.517100000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"a69cd7fa-c753-492c-af9c-eee806420850","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.3125037,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"a69cd7fa-c753-492c-af9c-eee806420850","ResponseTimeUtc":"2026-04-14T21:50:13.9956405Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: a69cd7fa-c753-492c-af9c-eee806420850, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150810","BELatencyInMs":"0.632"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 21:50:34 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5d303ee0-33ba-496d-9574-0be4d76f9e7e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5d303ee0-33ba-496d-9574-0be4d76f9e7e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5d303ee0-33ba-496d-9574-0be4d76f9e7e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5d303ee0-33ba-496d-9574-0be4d76f9e7e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:50:33.977Z","duration in milliseconds":48.620400000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:50:33.8524334Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94921,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0488},{"name":"Get PkValue From Stream","duration in milliseconds":6.5167,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.3638,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:50:33.9774376Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"52bb6c9e-8b70-43e0-828a-7aa6a9d14fae","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":41.8868,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":41.854600000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:49:39.6300968Z","cpu":2.188,"memory":5272124.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.102,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:49:49.6336025Z","cpu":0.547,"memory":5264744.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0792,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:49:59.6476698Z","cpu":2.808,"memory":5264120.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1129,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:50:09.6549407Z","cpu":3.900,"memory":5272540.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1002,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:50:19.6590551Z","cpu":4.844,"memory":5202008.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1295,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:50:29.6734530Z","cpu":3.432,"memory":5180300.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1047,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":41.8465,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":41.8415,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":41.8226,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":41.819700000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":40.9854,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:50:33.9931455Z","EndTimeUTC":"2026-04-14T21:50:33.9931455Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:50:34.0243104Z","DurationInMs":31.164900000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"5d303ee0-33ba-496d-9574-0be4d76f9e7e","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150810,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150810,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150810,"SessionToken":"-1#26150810","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.857","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:50:34 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:50:33.9931455Z", "durationInMs": 0.0087},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:50:33.9931542Z", "durationInMs": 22.0056},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:50:34.0151598Z", "durationInMs": 0.2077},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:50:34.0153675Z", "durationInMs": 1.9288},{"event": "Received", "startTimeUtc": "2026-04-14T21:50:34.0172963Z", "durationInMs": 0.135},{"event": "Completed", "startTimeUtc": "2026-04-14T21:50:34.0174313Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:50:34.0243104Z","lastSend":"2026-04-14T21:50:34.0243104Z","lastReceive":"2026-04-14T21:50:34.0243104Z"},"requestSizeInBytes":1633,"requestBodySizeInBytes":1183,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:50:33.9774376Z","DurationInMs":15.7079,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"0b14aef3-3592-46da-8431-b869f48c8fce","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:50:33.9931455Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"deb9d902-5624-4070-af61-b032e57fb903","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:50:33.9931455Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"5d303ee0-33ba-496d-9574-0be4d76f9e7e","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.171877,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"5d303ee0-33ba-496d-9574-0be4d76f9e7e","ResponseTimeUtc":"2026-04-14T21:50:34.0243104Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 5d303ee0-33ba-496d-9574-0be4d76f9e7e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150810","BELatencyInMs":"0.857"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 21:50:46 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5020d8b9-0b96-46d3-9193-4192d0e5f9a7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5020d8b9-0b96-46d3-9193-4192d0e5f9a7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5020d8b9-0b96-46d3-9193-4192d0e5f9a7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5020d8b9-0b96-46d3-9193-4192d0e5f9a7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:50:46.437Z","duration in milliseconds":44.763000000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:50:46.3427212Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94925,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0468},{"name":"Get PkValue From Stream","duration in milliseconds":7.9779,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":7.8367,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:50:46.4371766Z","DurationInMs":15.7285,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"13e04c3d-7906-4924-9dc1-bd508f30120a","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.5689,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.5364,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:49:49.6336025Z","cpu":0.547,"memory":5264744.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0792,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:49:59.6476698Z","cpu":2.808,"memory":5264120.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1129,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:50:09.6549407Z","cpu":3.900,"memory":5272540.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1002,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:50:19.6590551Z","cpu":4.844,"memory":5202008.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1295,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:50:29.6734530Z","cpu":3.432,"memory":5180300.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1047,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:50:39.6786595Z","cpu":3.359,"memory":5183756.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1178,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.529900000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.5248,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.504400000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.501400000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.5871,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:50:46.4529051Z","EndTimeUTC":"2026-04-14T21:50:46.4684514Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:50:46.4841491Z","DurationInMs":15.697700000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"5020d8b9-0b96-46d3-9193-4192d0e5f9a7","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150815,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150815,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150815,"SessionToken":"-1#26150815","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.547","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:50:46 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:50:46.4684514Z", "durationInMs": 0.0079},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:50:46.4684593Z", "durationInMs": 18.246},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:50:46.4867053Z", "durationInMs": 0.1907},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:50:46.4868960Z", "durationInMs": 1.6376},{"event": "Received", "startTimeUtc": "2026-04-14T21:50:46.4885336Z", "durationInMs": 0.1389},{"event": "Completed", "startTimeUtc": "2026-04-14T21:50:46.4886725Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:50:46.4841491Z","lastSend":"2026-04-14T21:50:46.4841491Z","lastReceive":"2026-04-14T21:50:46.4841491Z"},"requestSizeInBytes":1631,"requestBodySizeInBytes":1183,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:50:46.4529051Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"fbf7c044-6a13-49cd-b444-922fcc650375","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:50:46.4529051Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"8c1e55c7-b6e6-41e1-9a34-ee52b51e7922","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:50:46.4529051Z","DurationInMs":15.5463,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"5020d8b9-0b96-46d3-9193-4192d0e5f9a7","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.157053,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"5020d8b9-0b96-46d3-9193-4192d0e5f9a7","ResponseTimeUtc":"2026-04-14T21:50:46.4841491Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 5020d8b9-0b96-46d3-9193-4192d0e5f9a7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150815","BELatencyInMs":"0.547"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}
2026-04-14 21:51:55 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d56ef908-82a7-4515-bfdd-aeb9bd55c723; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d56ef908-82a7-4515-bfdd-aeb9bd55c723, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d56ef908-82a7-4515-bfdd-aeb9bd55c723; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d56ef908-82a7-4515-bfdd-aeb9bd55c723, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:51:55.461Z","duration in milliseconds":48.130900000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:51:55.2113732Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94933,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.046200000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":7.2075000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":7.0563,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:51:55.4613965Z","DurationInMs":15.6027,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"e9c3625b-7f6f-48ba-8147-948ea4946bd5","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":40.692800000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":40.6577,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:50:59.6944103Z","cpu":2.031,"memory":5214880.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1184,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:51:09.6983589Z","cpu":6.250,"memory":5160620.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0804,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:51:19.7024135Z","cpu":2.262,"memory":5159796.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0752,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:51:29.7041463Z","cpu":2.344,"memory":5159452.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0886,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:51:39.7084217Z","cpu":1.250,"memory":5160116.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0863,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:51:49.7194970Z","cpu":0.468,"memory":5161188.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2699,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":40.65,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":40.6456,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":40.624,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":40.6209,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":39.749500000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:51:55.4769992Z","EndTimeUTC":"2026-04-14T21:51:55.4926212Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:51:55.5083188Z","DurationInMs":15.697600000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"d56ef908-82a7-4515-bfdd-aeb9bd55c723","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150819,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150819,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150819,"SessionToken":"-1#26150819","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"1.026","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:51:55 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:51:55.4926212Z", "durationInMs": 0.0096},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:51:55.4926308Z", "durationInMs": 21.821},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:51:55.5144518Z", "durationInMs": 0.2231},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:51:55.5146749Z", "durationInMs": 2.7308},{"event": "Received", "startTimeUtc": "2026-04-14T21:51:55.5174057Z", "durationInMs": 0.1531},{"event": "Completed", "startTimeUtc": "2026-04-14T21:51:55.5175588Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:51:55.5083188Z","lastSend":"2026-04-14T21:51:55.5083188Z","lastReceive":"2026-04-14T21:51:55.5083188Z"},"requestSizeInBytes":1602,"requestBodySizeInBytes":1152,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:51:55.4769992Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"4e934506-d51c-43d4-9deb-4c104f9dd476","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:51:55.4769992Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"8017da27-fbff-4d34-bdc5-25ac6255af9d","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:51:55.4769992Z","DurationInMs":15.622,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"d56ef908-82a7-4515-bfdd-aeb9bd55c723","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2969456,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"d56ef908-82a7-4515-bfdd-aeb9bd55c723","ResponseTimeUtc":"2026-04-14T21:51:55.5083188Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: d56ef908-82a7-4515-bfdd-aeb9bd55c723, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150819","BELatencyInMs":"1.026"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 21:52:52 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: ac7cdc18-7ed8-4bcd-a63c-e07bc820e2f1; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: ac7cdc18-7ed8-4bcd-a63c-e07bc820e2f1, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: ac7cdc18-7ed8-4bcd-a63c-e07bc820e2f1; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: ac7cdc18-7ed8-4bcd-a63c-e07bc820e2f1, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:52:52.272Z","duration in milliseconds":43.941700000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:52:51.9446372Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94941,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0461},{"name":"Get PkValue From Stream","duration in milliseconds":6.2328,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.0599000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:52:52.2727758Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"41cde311-812b-4c6a-b1ba-a6e13a43e1f3","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.4927,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.4565,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:51:59.7305510Z","cpu":2.969,"memory":5158508.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1066,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:52:09.7398423Z","cpu":3.978,"memory":5146644.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1044,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:52:19.7461851Z","cpu":2.656,"memory":5152484.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0887,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:52:29.7555530Z","cpu":3.978,"memory":5169940.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0379,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:52:39.7716237Z","cpu":1.560,"memory":5167100.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0885,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:52:49.7884667Z","cpu":1.014,"memory":5170628.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1855,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.4498,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.4454,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.4244,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.4212,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.4716,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:52:52.2883973Z","EndTimeUTC":"2026-04-14T21:52:52.2883973Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:52:52.3196444Z","DurationInMs":31.247100000000003,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"ac7cdc18-7ed8-4bcd-a63c-e07bc820e2f1","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150821,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150821,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150821,"SessionToken":"-1#26150821","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.709","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:52:52 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:52:52.2883973Z", "durationInMs": 0.0081},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:52:52.2884054Z", "durationInMs": 19.8455},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:52:52.3082509Z", "durationInMs": 0.1852},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:52:52.3084361Z", "durationInMs": 1.8603},{"event": "Received", "startTimeUtc": "2026-04-14T21:52:52.3102964Z", "durationInMs": 0.1247},{"event": "Completed", "startTimeUtc": "2026-04-14T21:52:52.3104211Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:52:52.3040284Z","lastSend":"2026-04-14T21:52:52.3040284Z","lastReceive":"2026-04-14T21:52:52.3196444Z"},"requestSizeInBytes":1637,"requestBodySizeInBytes":1183,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:52:52.2883973Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"a4e97452-1468-4762-bcb5-19044426956d","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:52:52.2883973Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"9c0d50c3-d6ca-42a9-bcad-660bc50f3452","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:52:52.2883973Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"ac7cdc18-7ed8-4bcd-a63c-e07bc820e2f1","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.3750072,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"ac7cdc18-7ed8-4bcd-a63c-e07bc820e2f1","ResponseTimeUtc":"2026-04-14T21:52:52.3196444Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: ac7cdc18-7ed8-4bcd-a63c-e07bc820e2f1, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150821","BELatencyInMs":"0.709"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 21:53:47 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: df5aeaee-ea73-48ff-96b2-1c00b1ac2c16; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: df5aeaee-ea73-48ff-96b2-1c00b1ac2c16, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: df5aeaee-ea73-48ff-96b2-1c00b1ac2c16; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: df5aeaee-ea73-48ff-96b2-1c00b1ac2c16, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:53:47.497Z","duration in milliseconds":41.9255,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:53:47.3566883Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94949,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.038900000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":6.0088,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014},{"name":"Read Collection","duration in milliseconds":5.7688000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:53:47.4973048Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"496c2a2f-18e4-42db-b979-db0ec21dfea1","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.6983,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.6567,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:52:49.7884667Z","cpu":1.014,"memory":5170628.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1855,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:52:59.8034859Z","cpu":2.340,"memory":5288836.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0953,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:53:09.8142500Z","cpu":1.170,"memory":5274772.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.099,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:53:19.8185795Z","cpu":3.047,"memory":5271312.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0953,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:53:29.8199096Z","cpu":1.641,"memory":5266472.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0962,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:53:39.8290625Z","cpu":1.950,"memory":5267496.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0866,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.6483,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.6424,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.618500000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.6152,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.729800000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:53:47.5129274Z","EndTimeUTC":"2026-04-14T21:53:47.5129274Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:53:47.5285573Z","DurationInMs":15.629900000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"df5aeaee-ea73-48ff-96b2-1c00b1ac2c16","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150823,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150823,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150823,"SessionToken":"-1#26150823","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.666","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:53:47 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:53:47.5129274Z", "durationInMs": 0.0093},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:53:47.5129367Z", "durationInMs": 18.7774},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:53:47.5317141Z", "durationInMs": 0.197},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:53:47.5319111Z", "durationInMs": 1.9644},{"event": "Received", "startTimeUtc": "2026-04-14T21:53:47.5338755Z", "durationInMs": 0.1348},{"event": "Completed", "startTimeUtc": "2026-04-14T21:53:47.5340103Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:53:47.5285573Z","lastSend":"2026-04-14T21:53:47.5285573Z","lastReceive":"2026-04-14T21:53:47.5285573Z"},"requestSizeInBytes":1636,"requestBodySizeInBytes":1184,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:53:47.4973048Z","DurationInMs":15.6226,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"6931c5ee-9cf2-449c-9967-28cb16a3294c","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:53:47.5129274Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"5fafa783-89fc-4261-8a8b-c4f1751c299e","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:53:47.5129274Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"df5aeaee-ea73-48ff-96b2-1c00b1ac2c16","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1874961,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"df5aeaee-ea73-48ff-96b2-1c00b1ac2c16","ResponseTimeUtc":"2026-04-14T21:53:47.5441844Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: df5aeaee-ea73-48ff-96b2-1c00b1ac2c16, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150823","BELatencyInMs":"0.666"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]}
2026-04-14 21:53:57 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e31a358b-a35f-4372-9a10-a36019d72576; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e31a358b-a35f-4372-9a10-a36019d72576, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: e31a358b-a35f-4372-9a10-a36019d72576; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: e31a358b-a35f-4372-9a10-a36019d72576, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:53:57.845Z","duration in milliseconds":43.7521,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:53:57.7208809Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94952,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.059800000000000006},{"name":"Get PkValue From Stream","duration in milliseconds":7.0352000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0016},{"name":"Read Collection","duration in milliseconds":6.8562,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:53:57.8458854Z","DurationInMs":15.6286,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"3ded223d-04dc-4554-ae26-8c45af71f85e","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.4538,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.4119,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:52:59.8034859Z","cpu":2.340,"memory":5288836.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0953,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:53:09.8142500Z","cpu":1.170,"memory":5274772.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.099,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:53:19.8185795Z","cpu":3.047,"memory":5271312.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0953,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:53:29.8199096Z","cpu":1.641,"memory":5266472.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0962,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:53:39.8290625Z","cpu":1.950,"memory":5267496.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0866,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:53:49.8311636Z","cpu":1.797,"memory":5289432.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1032,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.4044,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.3984,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.3755,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.3726,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.4421,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:53:57.8615140Z","EndTimeUTC":"2026-04-14T21:53:57.8771351Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:53:57.8927617Z","DurationInMs":15.626600000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"e31a358b-a35f-4372-9a10-a36019d72576","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150823,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150823,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150823,"SessionToken":"-1#26150823","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.537","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:53:57 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:53:57.8771351Z", "durationInMs": 0.0076},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:53:57.8771427Z", "durationInMs": 17.3653},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:53:57.8945080Z", "durationInMs": 0.2261},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:53:57.8947341Z", "durationInMs": 2.7897},{"event": "Received", "startTimeUtc": "2026-04-14T21:53:57.8975238Z", "durationInMs": 0.185},{"event": "Completed", "startTimeUtc": "2026-04-14T21:53:57.8977088Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:53:57.8927617Z","lastSend":"2026-04-14T21:53:57.8927617Z","lastReceive":"2026-04-14T21:53:57.8927617Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:53:57.8615140Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"09992ddd-a9d8-406a-9377-1e425142c7b1","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:53:57.8615140Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"71dca6d3-cae4-4c73-8eb9-3f4290804528","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:53:57.8615140Z","DurationInMs":15.6211,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"e31a358b-a35f-4372-9a10-a36019d72576","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1875044,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"e31a358b-a35f-4372-9a10-a36019d72576","ResponseTimeUtc":"2026-04-14T21:53:57.8927617Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: e31a358b-a35f-4372-9a10-a36019d72576, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150823","BELatencyInMs":"0.537"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.003}]}
2026-04-14 21:54:26 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 989f5fc4-0dc4-4d58-8e30-e8f998bccf46; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 989f5fc4-0dc4-4d58-8e30-e8f998bccf46, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 989f5fc4-0dc4-4d58-8e30-e8f998bccf46; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 989f5fc4-0dc4-4d58-8e30-e8f998bccf46, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:54:26.710Z","duration in milliseconds":50.194500000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:54:26.4763743Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94958,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0524},{"name":"Get PkValue From Stream","duration in milliseconds":6.7197000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014},{"name":"Read Collection","duration in milliseconds":6.5667,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:54:26.7107565Z","DurationInMs":15.620800000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"5b3e3b64-a681-4757-af0e-3cdd2f36bd66","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":43.235600000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":43.2003,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:53:29.8199096Z","cpu":1.641,"memory":5266472.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0962,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:53:39.8290625Z","cpu":1.950,"memory":5267496.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0866,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:53:49.8311636Z","cpu":1.797,"memory":5289432.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1032,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:53:59.8364970Z","cpu":3.281,"memory":5293012.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.106,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:54:09.8406949Z","cpu":2.891,"memory":5285684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1088,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:54:19.8420800Z","cpu":3.750,"memory":5283224.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1213,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":43.193000000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":43.1892,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":43.1695,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":43.1666,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":42.1657,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:54:26.7420831Z","EndTimeUTC":"2026-04-14T21:54:26.7420831Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:54:26.7576262Z","DurationInMs":15.5431,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"989f5fc4-0dc4-4d58-8e30-e8f998bccf46","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150827,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150827,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150827,"SessionToken":"-1#26150827","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.635","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:54:26 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:54:26.7420831Z", "durationInMs": 0.0081},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:54:26.7420912Z", "durationInMs": 22.3891},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:54:26.7644803Z", "durationInMs": 0.2095},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:54:26.7646898Z", "durationInMs": 2.0968},{"event": "Received", "startTimeUtc": "2026-04-14T21:54:26.7667866Z", "durationInMs": 0.156},{"event": "Completed", "startTimeUtc": "2026-04-14T21:54:26.7669426Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:54:26.7576262Z","lastSend":"2026-04-14T21:54:26.7576262Z","lastReceive":"2026-04-14T21:54:26.7576262Z"},"requestSizeInBytes":1602,"requestBodySizeInBytes":1152,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:54:26.7263773Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"94cc82b1-0588-4940-b41b-cc419966c7ad","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:54:26.7263773Z","DurationInMs":15.7058,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"eac1660a-cd3c-4d61-b728-cf5f2f79f0dd","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:54:26.7420831Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"989f5fc4-0dc4-4d58-8e30-e8f998bccf46","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2968786,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"989f5fc4-0dc4-4d58-8e30-e8f998bccf46","ResponseTimeUtc":"2026-04-14T21:54:26.7732529Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 989f5fc4-0dc4-4d58-8e30-e8f998bccf46, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150827","BELatencyInMs":"0.635"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 21:54:46 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5f7bbbce-ade4-4d41-b938-dbaa7059e9f8; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5f7bbbce-ade4-4d41-b938-dbaa7059e9f8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5f7bbbce-ade4-4d41-b938-dbaa7059e9f8; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5f7bbbce-ade4-4d41-b938-dbaa7059e9f8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:54:46.277Z","duration in milliseconds":43.0486,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:54:46.0589598Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94967,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0359},{"name":"Get PkValue From Stream","duration in milliseconds":6.3934000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":6.2207,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:54:46.2777164Z","DurationInMs":15.6258,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"7a5569bb-59a1-4adb-bfff-e38a0e0dd846","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.2926,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.2536,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:53:49.8311636Z","cpu":1.797,"memory":5289432.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1032,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:53:59.8364970Z","cpu":3.281,"memory":5293012.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.106,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:54:09.8406949Z","cpu":2.891,"memory":5285684.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1088,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:54:19.8420800Z","cpu":3.750,"memory":5283224.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1213,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:54:29.8507039Z","cpu":2.418,"memory":5286652.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3919,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:54:39.8571834Z","cpu":3.594,"memory":5283048.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1072,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.2419,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.2354,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.2108,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.2065,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.937000000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:54:46.2933422Z","EndTimeUTC":"2026-04-14T21:54:46.3089659Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:54:46.3245928Z","DurationInMs":15.626900000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"5f7bbbce-ade4-4d41-b938-dbaa7059e9f8","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150830,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150830,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150830,"SessionToken":"-1#26150830","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.546","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:54:46 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:54:46.3089659Z", "durationInMs": 0.0146},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:54:46.3089805Z", "durationInMs": 16.4709},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:54:46.3254514Z", "durationInMs": 0.264},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:54:46.3257154Z", "durationInMs": 1.6865},{"event": "Received", "startTimeUtc": "2026-04-14T21:54:46.3274019Z", "durationInMs": 0.1713},{"event": "Completed", "startTimeUtc": "2026-04-14T21:54:46.3275732Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:54:46.3245928Z","lastSend":"2026-04-14T21:54:46.3245928Z","lastReceive":"2026-04-14T21:54:46.3245928Z"},"requestSizeInBytes":1635,"requestBodySizeInBytes":1183,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:54:46.2933422Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"5b0b6b1a-1550-4059-95db-d31ec5495a7d","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:54:46.2933422Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"417e9133-656b-4ec3-9254-8e1fabe4f9d8","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:54:46.2933422Z","DurationInMs":15.623700000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"5f7bbbce-ade4-4d41-b938-dbaa7059e9f8","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.265633,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"5f7bbbce-ade4-4d41-b938-dbaa7059e9f8","ResponseTimeUtc":"2026-04-14T21:54:46.3245928Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 5f7bbbce-ade4-4d41-b938-dbaa7059e9f8, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150830","BELatencyInMs":"0.546"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0026000000000000003}]}
2026-04-14 21:55:10 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: f7f7fd2e-8148-4b6f-820f-c10caf91f4bb; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: f7f7fd2e-8148-4b6f-820f-c10caf91f4bb, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: f7f7fd2e-8148-4b6f-820f-c10caf91f4bb; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: f7f7fd2e-8148-4b6f-820f-c10caf91f4bb, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:55:10.736Z","duration in milliseconds":71.028,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:55:10.5489649Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94970,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0551},{"name":"Get PkValue From Stream","duration in milliseconds":7.1430000000000007,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0044},{"name":"Read Collection","duration in milliseconds":6.9681000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:55:10.7365455Z","DurationInMs":15.6265,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"17cca990-d2bf-4af7-8dde-ec043a1ef263","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":63.551500000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":63.507400000000004,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:54:19.8420800Z","cpu":3.750,"memory":5283224.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1213,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:54:29.8507039Z","cpu":2.418,"memory":5286652.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3919,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:54:39.8571834Z","cpu":3.594,"memory":5283048.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1072,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:54:49.8588309Z","cpu":4.766,"memory":5277988.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0989,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:54:59.8721906Z","cpu":2.262,"memory":5279136.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1922,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:55:09.8771527Z","cpu":2.574,"memory":5268412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0987,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":63.496500000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":63.4913,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":63.4652,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":63.462300000000006,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":48.9902,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:55:10.7521720Z","EndTimeUTC":"2026-04-14T21:55:10.7677205Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:55:10.7989682Z","DurationInMs":31.247700000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"f7f7fd2e-8148-4b6f-820f-c10caf91f4bb","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150830,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150830,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150830,"SessionToken":"-1#26150830","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.986","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:55:10 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:55:10.7677205Z", "durationInMs": 0.0081},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:55:10.7677286Z", "durationInMs": 30.3275},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:55:10.7980561Z", "durationInMs": 0.2052},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:55:10.7982613Z", "durationInMs": 2.3604},{"event": "Received", "startTimeUtc": "2026-04-14T21:55:10.8006217Z", "durationInMs": 0.1553},{"event": "Completed", "startTimeUtc": "2026-04-14T21:55:10.8007770Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:55:10.7833657Z","lastSend":"2026-04-14T21:55:10.7833657Z","lastReceive":"2026-04-14T21:55:10.7989682Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:55:10.7521720Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"509b06a5-66f5-4f09-b093-e16853b2724c","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:55:10.7521720Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"88cb3435-0680-4427-9a41-4751539815b6","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:55:10.7521720Z","DurationInMs":15.5485,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"f7f7fd2e-8148-4b6f-820f-c10caf91f4bb","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2500033,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"f7f7fd2e-8148-4b6f-820f-c10caf91f4bb","ResponseTimeUtc":"2026-04-14T21:55:10.8145983Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: f7f7fd2e-8148-4b6f-820f-c10caf91f4bb, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150830","BELatencyInMs":"0.986"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0024000000000000002}]}
2026-04-14 21:56:08 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 47417015-606a-43b8-a1ed-553ce0926eae; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 47417015-606a-43b8-a1ed-553ce0926eae, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 47417015-606a-43b8-a1ed-553ce0926eae; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 47417015-606a-43b8-a1ed-553ce0926eae, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:56:08.475Z","duration in milliseconds":42.3588,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:56:08.2876523Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":94982,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.046700000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":6.9787,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0015},{"name":"Read Collection","duration in milliseconds":6.8211,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:56:08.4752257Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"e1a10a33-08f4-4df5-81ae-89d89103e78d","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.1529,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.1165,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:55:09.8771527Z","cpu":2.574,"memory":5268412.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0987,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:55:19.8834479Z","cpu":2.422,"memory":5252688.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1055,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:55:29.8960528Z","cpu":2.340,"memory":5234780.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0803,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:55:39.9014571Z","cpu":2.656,"memory":5234332.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2682,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:55:49.9109817Z","cpu":2.808,"memory":5236528.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1037,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:55:59.9123705Z","cpu":2.891,"memory":5196272.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0963,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.1081,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.1042,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.0812,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.0783,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.2058,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:56:08.4908277Z","EndTimeUTC":"2026-04-14T21:56:08.4908277Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:56:08.5064023Z","DurationInMs":15.5746,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"47417015-606a-43b8-a1ed-553ce0926eae","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150836,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150836,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150836,"SessionToken":"-1#26150836","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.656","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:56:08 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:56:08.4908277Z", "durationInMs": 0.0097},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:56:08.4908374Z", "durationInMs": 17.482},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:56:08.5083194Z", "durationInMs": 0.2075},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:56:08.5085269Z", "durationInMs": 1.8053},{"event": "Received", "startTimeUtc": "2026-04-14T21:56:08.5103322Z", "durationInMs": 0.1421},{"event": "Completed", "startTimeUtc": "2026-04-14T21:56:08.5104743Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:56:08.5064023Z","lastSend":"2026-04-14T21:56:08.5064023Z","lastReceive":"2026-04-14T21:56:08.5064023Z"},"requestSizeInBytes":1602,"requestBodySizeInBytes":1152,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:56:08.4752257Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"aaf19404-2cd0-4c36-9436-23634d3e654c","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:56:08.4752257Z","DurationInMs":15.602,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"bf092d05-5b73-49e6-862e-b13a076977b6","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:56:08.4908277Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"47417015-606a-43b8-a1ed-553ce0926eae","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.21875,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"47417015-606a-43b8-a1ed-553ce0926eae","ResponseTimeUtc":"2026-04-14T21:56:08.5064023Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 47417015-606a-43b8-a1ed-553ce0926eae, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150836","BELatencyInMs":"0.656"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 21:57:12 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 247fcdc2-4288-478d-9879-3c660104837e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 247fcdc2-4288-478d-9879-3c660104837e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 247fcdc2-4288-478d-9879-3c660104837e; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 247fcdc2-4288-478d-9879-3c660104837e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:57:12.582Z","duration in milliseconds":55.3049,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:57:12.4575979Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95015,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0555},{"name":"Get PkValue From Stream","duration in milliseconds":10.3741,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0016},{"name":"Read Collection","duration in milliseconds":10.1767,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:57:12.5826019Z","DurationInMs":15.625200000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"49b1321f-a986-4108-841c-8c55a5565f60","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":44.676700000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":44.6312,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:56:19.9406417Z","cpu":0.390,"memory":5201144.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.092,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:56:29.9556877Z","cpu":3.125,"memory":5197168.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1066,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:56:39.9698347Z","cpu":3.120,"memory":5197188.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2305,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:56:49.9868186Z","cpu":0.936,"memory":5211192.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.261,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:56:59.9967352Z","cpu":3.042,"memory":5209772.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1208,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:57:10.0098648Z","cpu":7.098,"memory":5194236.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3102,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":44.6216,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":44.618,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":44.594500000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":44.5921,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":43.6133,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:57:12.5982271Z","EndTimeUTC":"2026-04-14T21:57:12.6139177Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:57:12.6295597Z","DurationInMs":15.642000000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"247fcdc2-4288-478d-9879-3c660104837e","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150837,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150837,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150837,"SessionToken":"-1#26150837","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.625","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:57:12 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:57:12.6139177Z", "durationInMs": 0.0084},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:57:12.6139261Z", "durationInMs": 23.4084},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:57:12.6373345Z", "durationInMs": 0.2102},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:57:12.6375447Z", "durationInMs": 3.9947},{"event": "Received", "startTimeUtc": "2026-04-14T21:57:12.6415394Z", "durationInMs": 0.1576},{"event": "Completed", "startTimeUtc": "2026-04-14T21:57:12.6416970Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:57:12.6295597Z","lastSend":"2026-04-14T21:57:12.6295597Z","lastReceive":"2026-04-14T21:57:12.6295597Z"},"requestSizeInBytes":1635,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:57:12.5982271Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"f53110ef-49b2-4fc5-9a0e-3e07dfb5a02b","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:57:12.5982271Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"9e0ae7ab-ca52-4558-90e1-56e17c25a98d","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:57:12.5982271Z","DurationInMs":15.6906,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"247fcdc2-4288-478d-9879-3c660104837e","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1719618,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"247fcdc2-4288-478d-9879-3c660104837e","ResponseTimeUtc":"2026-04-14T21:57:12.6295597Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 247fcdc2-4288-478d-9879-3c660104837e, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150837","BELatencyInMs":"0.625"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.003}]}
2026-04-14 21:57:29 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 58c79d65-4119-4036-89f7-adfc789b57d7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 58c79d65-4119-4036-89f7-adfc789b57d7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 58c79d65-4119-4036-89f7-adfc789b57d7; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 58c79d65-4119-4036-89f7-adfc789b57d7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:57:29.801Z","duration in milliseconds":44.7245,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:57:29.4110495Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95022,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0443},{"name":"Get PkValue From Stream","duration in milliseconds":7.6503000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0012000000000000001},{"name":"Read Collection","duration in milliseconds":7.5035000000000007,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:57:29.8016779Z","DurationInMs":15.7126,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"a08ad0f5-4a9f-4224-a181-0ae01d165474","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.8589,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.8282,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:56:29.9556877Z","cpu":3.125,"memory":5197168.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1066,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:56:39.9698347Z","cpu":3.120,"memory":5197188.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2305,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:56:49.9868186Z","cpu":0.936,"memory":5211192.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.261,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:56:59.9967352Z","cpu":3.042,"memory":5209772.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1208,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:57:10.0098648Z","cpu":7.098,"memory":5194236.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3102,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:57:20.0174032Z","cpu":7.344,"memory":5193464.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1136,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.8213,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.8162,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.7956,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.7935,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.8502,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:57:29.8173905Z","EndTimeUTC":"2026-04-14T21:57:29.8173905Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:57:29.8485534Z","DurationInMs":31.1629,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"58c79d65-4119-4036-89f7-adfc789b57d7","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150840,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150840,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150840,"SessionToken":"-1#26150840","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.532","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:57:29 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:57:29.8173905Z", "durationInMs": 0.0079},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:57:29.8173984Z", "durationInMs": 18.893},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:57:29.8362914Z", "durationInMs": 0.2025},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:57:29.8364939Z", "durationInMs": 1.8489},{"event": "Received", "startTimeUtc": "2026-04-14T21:57:29.8383428Z", "durationInMs": 0.1382},{"event": "Completed", "startTimeUtc": "2026-04-14T21:57:29.8384810Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:57:29.8485534Z","lastSend":"2026-04-14T21:57:29.8485534Z","lastReceive":"2026-04-14T21:57:29.8485534Z"},"requestSizeInBytes":1632,"requestBodySizeInBytes":1180,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:57:29.8173905Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"fa0edcbb-97fc-42a7-8eaf-2021566b77aa","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:57:29.8173905Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"99ebe113-7d3e-4742-8328-88e9f9767cf9","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:57:29.8173905Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"58c79d65-4119-4036-89f7-adfc789b57d7","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.4375039,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"58c79d65-4119-4036-89f7-adfc789b57d7","ResponseTimeUtc":"2026-04-14T21:57:29.8485534Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 58c79d65-4119-4036-89f7-adfc789b57d7, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150840","BELatencyInMs":"0.532"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 21:57:31 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d2c0d756-40b3-42cc-9642-e54f9674f891; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d2c0d756-40b3-42cc-9642-e54f9674f891, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: d2c0d756-40b3-42cc-9642-e54f9674f891; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: d2c0d756-40b3-42cc-9642-e54f9674f891, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:57:31.511Z","duration in milliseconds":41.1633,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:57:31.4175079Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95023,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.056100000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":6.3678,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0014},{"name":"Read Collection","duration in milliseconds":6.1985,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:57:31.5113466Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"5fba22cc-887d-4421-86c0-dc3f367359a9","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":34.556000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":34.5188,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:56:39.9698347Z","cpu":3.120,"memory":5197188.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2305,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:56:49.9868186Z","cpu":0.936,"memory":5211192.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.261,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:56:59.9967352Z","cpu":3.042,"memory":5209772.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1208,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:57:10.0098648Z","cpu":7.098,"memory":5194236.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3102,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:57:20.0174032Z","cpu":7.344,"memory":5193464.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1136,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:57:30.0205098Z","cpu":1.248,"memory":5193856.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1025,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":34.5112,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":34.5038,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":34.4823,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":34.4797,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":33.585100000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:57:31.5268779Z","EndTimeUTC":"2026-04-14T21:57:31.5268779Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:57:31.5581284Z","DurationInMs":31.250500000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"d2c0d756-40b3-42cc-9642-e54f9674f891","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150840,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150840,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150840,"SessionToken":"-1#26150840","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.639","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:57:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:57:31.5268779Z", "durationInMs": 0.0083},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:57:31.5268862Z", "durationInMs": 15.9862},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:57:31.5428724Z", "durationInMs": 0.2466},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:57:31.5431190Z", "durationInMs": 2.2811},{"event": "Received", "startTimeUtc": "2026-04-14T21:57:31.5454001Z", "durationInMs": 0.1618},{"event": "Completed", "startTimeUtc": "2026-04-14T21:57:31.5455619Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:57:31.5425043Z","lastSend":"2026-04-14T21:57:31.5425043Z","lastReceive":"2026-04-14T21:57:31.5425043Z"},"requestSizeInBytes":1633,"requestBodySizeInBytes":1185,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:57:31.5113466Z","DurationInMs":15.5313,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"00210c12-8221-4bd9-9dfe-f3b5359822f3","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:57:31.5268779Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"cc743cda-2cbb-4e97-ad8c-882219988004","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:57:31.5268779Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"d2c0d756-40b3-42cc-9642-e54f9674f891","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.14062049999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"d2c0d756-40b3-42cc-9642-e54f9674f891","ResponseTimeUtc":"2026-04-14T21:57:31.5581284Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: d2c0d756-40b3-42cc-9642-e54f9674f891, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150840","BELatencyInMs":"0.639"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 21:57:50 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 2a8e1f79-50a3-4505-a9b3-ae4de2089ff2; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 2a8e1f79-50a3-4505-a9b3-ae4de2089ff2, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 2a8e1f79-50a3-4505-a9b3-ae4de2089ff2; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 2a8e1f79-50a3-4505-a9b3-ae4de2089ff2, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:57:50.717Z","duration in milliseconds":46.7974,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:57:50.4052455Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95025,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0492},{"name":"Get PkValue From Stream","duration in milliseconds":6.3329,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.00090000000000000008},{"name":"Read Collection","duration in milliseconds":6.1867,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:57:50.7177651Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"23448742-0206-4de2-81f2-667764b00e73","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":40.233200000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":40.2027,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:56:59.9967352Z","cpu":3.042,"memory":5209772.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1208,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:57:10.0098648Z","cpu":7.098,"memory":5194236.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.3102,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:57:20.0174032Z","cpu":7.344,"memory":5193464.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1136,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:57:30.0205098Z","cpu":1.248,"memory":5193856.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1025,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:57:40.0282117Z","cpu":0.938,"memory":5193280.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0988,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:57:50.0390616Z","cpu":0.936,"memory":5214904.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2695,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":40.1952,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":40.1911,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":40.168400000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":40.164300000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":39.315400000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:57:50.7334265Z","EndTimeUTC":"2026-04-14T21:57:50.7334265Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:57:50.7646286Z","DurationInMs":31.2021,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"2a8e1f79-50a3-4505-a9b3-ae4de2089ff2","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150840,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150840,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150840,"SessionToken":"-1#26150840","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.629","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:57:50 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:57:50.7334265Z", "durationInMs": 0.0068},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:57:50.7334333Z", "durationInMs": 21.1174},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:57:50.7545507Z", "durationInMs": 0.2347},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:57:50.7547854Z", "durationInMs": 1.986},{"event": "Received", "startTimeUtc": "2026-04-14T21:57:50.7567714Z", "durationInMs": 0.158},{"event": "Completed", "startTimeUtc": "2026-04-14T21:57:50.7569294Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:57:50.7490015Z","lastSend":"2026-04-14T21:57:50.7490015Z","lastReceive":"2026-04-14T21:57:50.7646286Z"},"requestSizeInBytes":1634,"requestBodySizeInBytes":1182,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:57:50.7177651Z","DurationInMs":15.6614,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"23f88d1e-4fc4-45a3-9c36-7c64f8760dd5","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:57:50.7334265Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"af47a154-17db-44ac-a1cc-57d565b62e2d","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:57:50.7334265Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"2a8e1f79-50a3-4505-a9b3-ae4de2089ff2","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.3593831,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"2a8e1f79-50a3-4505-a9b3-ae4de2089ff2","ResponseTimeUtc":"2026-04-14T21:57:50.7646286Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 2a8e1f79-50a3-4505-a9b3-ae4de2089ff2, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150840","BELatencyInMs":"0.629"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0015}]}
2026-04-14 21:58:14 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 8ed886a9-85db-4173-825f-e9648b669e48; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 8ed886a9-85db-4173-825f-e9648b669e48, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 8ed886a9-85db-4173-825f-e9648b669e48; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 8ed886a9-85db-4173-825f-e9648b669e48, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:58:14.755Z","duration in milliseconds":42.397800000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:58:13.5049294Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95031,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.058800000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":6.5604000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0019},{"name":"Read Collection","duration in milliseconds":6.3871,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:58:14.7550318Z","DurationInMs":15.625900000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"4d8442da-3665-41ae-ab6b-f972ac1cd846","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":35.5935,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":35.5604,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:57:20.0174032Z","cpu":7.344,"memory":5193464.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1136,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:57:30.0205098Z","cpu":1.248,"memory":5193856.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1025,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:57:40.0282117Z","cpu":0.938,"memory":5193280.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0988,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:57:50.0390616Z","cpu":0.936,"memory":5214904.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2695,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:58:00.0506967Z","cpu":2.422,"memory":5293236.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1042,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:58:10.0613870Z","cpu":3.432,"memory":5264288.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0803,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":35.5518,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":35.548300000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":35.525800000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":35.5233,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.574400000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:58:14.7706577Z","EndTimeUTC":"2026-04-14T21:58:14.7862634Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:58:14.8018332Z","DurationInMs":15.5698,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"8ed886a9-85db-4173-825f-e9648b669e48","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150842,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150842,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150842,"SessionToken":"-1#26150842","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.586","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:58:14 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:58:14.7862634Z", "durationInMs": 0.0095},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:58:14.7862729Z", "durationInMs": 18.1373},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:58:14.8044102Z", "durationInMs": 0.1987},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:58:14.8046089Z", "durationInMs": 2.2421},{"event": "Received", "startTimeUtc": "2026-04-14T21:58:14.8068510Z", "durationInMs": 0.1355},{"event": "Completed", "startTimeUtc": "2026-04-14T21:58:14.8069865Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:58:14.8018332Z","lastSend":"2026-04-14T21:58:14.8018332Z","lastReceive":"2026-04-14T21:58:14.8018332Z"},"requestSizeInBytes":1626,"requestBodySizeInBytes":1176,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:58:14.7706577Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"e857da04-466c-4723-b034-b2a9aeb69831","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:58:14.7706577Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"8906e7ba-199b-4a34-bc2a-d00abb162b45","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:58:14.7706577Z","DurationInMs":15.6057,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"8ed886a9-85db-4173-825f-e9648b669e48","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":1.2969038,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"8ed886a9-85db-4173-825f-e9648b669e48","ResponseTimeUtc":"2026-04-14T21:58:14.8018332Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 8ed886a9-85db-4173-825f-e9648b669e48, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150842","BELatencyInMs":"0.586"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0016}]}
2026-04-14 21:58:42 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: bd165f44-eb38-411a-bdd5-15f9fc89b5f0; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: bd165f44-eb38-411a-bdd5-15f9fc89b5f0, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: bd165f44-eb38-411a-bdd5-15f9fc89b5f0; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: bd165f44-eb38-411a-bdd5-15f9fc89b5f0, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:58:42.476Z","duration in milliseconds":43.4257,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:58:42.3982932Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95035,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.056400000000000006},{"name":"Get PkValue From Stream","duration in milliseconds":6.6013,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003},{"name":"Read Collection","duration in milliseconds":6.4359,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:58:42.4760643Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"e586760f-599b-45fb-b330-ee459d37ecc4","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.576100000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.5387,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:57:50.0390616Z","cpu":0.936,"memory":5214904.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2695,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:58:00.0506967Z","cpu":2.422,"memory":5293236.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1042,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:58:10.0613870Z","cpu":3.432,"memory":5264288.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0803,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:58:20.0752048Z","cpu":7.956,"memory":5258556.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1271,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:58:30.0825718Z","cpu":3.516,"memory":5268988.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0831,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:58:40.0960731Z","cpu":1.014,"memory":5265420.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0716,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.5302,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.5245,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.5037,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.5005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":35.621300000000005,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:58:42.4917789Z","EndTimeUTC":"2026-04-14T21:58:42.4917789Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:58:42.5229396Z","DurationInMs":31.160700000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"bd165f44-eb38-411a-bdd5-15f9fc89b5f0","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150846,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150846,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150846,"SessionToken":"-1#26150846","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"1.029","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:58:42 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:58:42.4917789Z", "durationInMs": 0.0098},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:58:42.4917887Z", "durationInMs": 18.5126},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:58:42.5103013Z", "durationInMs": 0.2156},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:58:42.5105169Z", "durationInMs": 2.622},{"event": "Received", "startTimeUtc": "2026-04-14T21:58:42.5131389Z", "durationInMs": 0.1568},{"event": "Completed", "startTimeUtc": "2026-04-14T21:58:42.5132957Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:58:42.5073141Z","lastSend":"2026-04-14T21:58:42.5073141Z","lastReceive":"2026-04-14T21:58:42.5073141Z"},"requestSizeInBytes":1631,"requestBodySizeInBytes":1183,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:58:42.4760643Z","DurationInMs":15.7146,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"9b9c94f6-8cf5-426a-82e4-0211184cd70d","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:58:42.4917789Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"dfb5192e-847b-4973-8d89-810306e82079","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:58:42.4917789Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"bd165f44-eb38-411a-bdd5-15f9fc89b5f0","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.12464639999999999,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"bd165f44-eb38-411a-bdd5-15f9fc89b5f0","ResponseTimeUtc":"2026-04-14T21:58:42.5229396Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: bd165f44-eb38-411a-bdd5-15f9fc89b5f0, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150846","BELatencyInMs":"1.029"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 21:59:15 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: fb5bc258-9fba-48dc-a1c7-6e07bc775058; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: fb5bc258-9fba-48dc-a1c7-6e07bc775058, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: fb5bc258-9fba-48dc-a1c7-6e07bc775058; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: fb5bc258-9fba-48dc-a1c7-6e07bc775058, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:59:15.572Z","duration in milliseconds":39.0811,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:59:15.3066941Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95043,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0412},{"name":"Get PkValue From Stream","duration in milliseconds":5.8637000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.001},{"name":"Read Collection","duration in milliseconds":5.683,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:59:15.5723229Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"a1ad619a-7c8f-4c1e-ad83-b6ae7e5df201","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":32.999,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":32.9662,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:58:20.0752048Z","cpu":7.956,"memory":5258556.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1271,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:58:30.0825718Z","cpu":3.516,"memory":5268988.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0831,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:58:40.0960731Z","cpu":1.014,"memory":5265420.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0716,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:58:50.1015574Z","cpu":1.716,"memory":5262912.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1025,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:59:00.1096634Z","cpu":11.875,"memory":5209668.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.115,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:59:10.1258890Z","cpu":2.652,"memory":5194080.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0967,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":32.9592,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":32.9546,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":32.9332,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":32.9307,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":31.886000000000003,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:59:15.5879500Z","EndTimeUTC":"2026-04-14T21:59:15.5879500Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:59:15.6035750Z","DurationInMs":15.625,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"fb5bc258-9fba-48dc-a1c7-6e07bc775058","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150848,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150848,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150848,"SessionToken":"-1#26150848","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.798","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:59:15 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:59:15.5879500Z", "durationInMs": 0.0087},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:59:15.5879587Z", "durationInMs": 15.5519},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:59:15.6035106Z", "durationInMs": 0.237},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:59:15.6037476Z", "durationInMs": 2.0481},{"event": "Received", "startTimeUtc": "2026-04-14T21:59:15.6057957Z", "durationInMs": 0.1589},{"event": "Completed", "startTimeUtc": "2026-04-14T21:59:15.6059546Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:59:15.6035750Z","lastSend":"2026-04-14T21:59:15.6035750Z","lastReceive":"2026-04-14T21:59:15.6035750Z"},"requestSizeInBytes":1600,"requestBodySizeInBytes":1152,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:59:15.5723229Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"ff8163e4-fac1-47f9-bcf3-7b481247dc73","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:59:15.5723229Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"09f8ac49-ec99-4551-8d4d-dd4ff25bbf0b","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:59:15.5879500Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"fb5bc258-9fba-48dc-a1c7-6e07bc775058","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2968809,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"fb5bc258-9fba-48dc-a1c7-6e07bc775058","ResponseTimeUtc":"2026-04-14T21:59:15.6035750Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: fb5bc258-9fba-48dc-a1c7-6e07bc775058, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150848","BELatencyInMs":"0.798"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0014}]}
2026-04-14 21:59:46 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5ff80e08-6d42-46ed-bacd-5af32b2a1dd3; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5ff80e08-6d42-46ed-bacd-5af32b2a1dd3, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 5ff80e08-6d42-46ed-bacd-5af32b2a1dd3; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 5ff80e08-6d42-46ed-bacd-5af32b2a1dd3, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T21:59:46.610Z","duration in milliseconds":49.5844,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T21:59:46.5175017Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95046,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.052500000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":6.1739000000000006,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0013000000000000002},{"name":"Read Collection","duration in milliseconds":6.0164,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:59:46.6108688Z","DurationInMs":15.609300000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"05bacf3a-c6be-4e20-ab56-a77ff3587dae","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":43.0719,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":43.0377,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:58:50.1015574Z","cpu":1.716,"memory":5262912.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1025,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:59:00.1096634Z","cpu":11.875,"memory":5209668.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.115,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:59:10.1258890Z","cpu":2.652,"memory":5194080.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0967,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:59:20.1274262Z","cpu":3.984,"memory":5191744.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1061,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:59:30.1321959Z","cpu":2.184,"memory":5187212.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0989,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:59:40.1375383Z","cpu":12.266,"memory":5219916.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2751,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":43.025800000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":43.0221,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":43.0007,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":42.998000000000005,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":42.0232,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T21:59:46.6264781Z","EndTimeUTC":"2026-04-14T21:59:46.6420310Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T21:59:46.6577325Z","DurationInMs":15.701500000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"5ff80e08-6d42-46ed-bacd-5af32b2a1dd3","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150849,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150849,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150849,"SessionToken":"-1#26150849","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.683","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 9:59:46 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T21:59:46.6420310Z", "durationInMs": 0.0098},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T21:59:46.6420408Z", "durationInMs": 25.4568},{"event": "Pipelined", "startTimeUtc": "2026-04-14T21:59:46.6674976Z", "durationInMs": 0.2044},{"event": "Transit Time", "startTimeUtc": "2026-04-14T21:59:46.6677020Z", "durationInMs": 1.8683},{"event": "Received", "startTimeUtc": "2026-04-14T21:59:46.6695703Z", "durationInMs": 0.127},{"event": "Completed", "startTimeUtc": "2026-04-14T21:59:46.6696973Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T21:59:46.6577325Z","lastSend":"2026-04-14T21:59:46.6577325Z","lastReceive":"2026-04-14T21:59:46.6577325Z"},"requestSizeInBytes":1604,"requestBodySizeInBytes":1152,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T21:59:46.6264781Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"64479a8b-3590-48b4-826d-51ff3bee3810","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T21:59:46.6264781Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"65004143-651b-44b1-8fa2-8c96efff58a9","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T21:59:46.6264781Z","DurationInMs":15.552900000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"5ff80e08-6d42-46ed-bacd-5af32b2a1dd3","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1563307,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"5ff80e08-6d42-46ed-bacd-5af32b2a1dd3","ResponseTimeUtc":"2026-04-14T21:59:46.6577325Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 5ff80e08-6d42-46ed-bacd-5af32b2a1dd3, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150849","BELatencyInMs":"0.683"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 22:00:18 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 2a8f8799-4579-4c91-987c-b2e867a8d66a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 2a8f8799-4579-4c91-987c-b2e867a8d66a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 2a8f8799-4579-4c91-987c-b2e867a8d66a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 2a8f8799-4579-4c91-987c-b2e867a8d66a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T22:00:18.448Z","duration in milliseconds":44.0732,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T22:00:18.2294651Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95053,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0398},{"name":"Get PkValue From Stream","duration in milliseconds":6.4783,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":6.3256000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T22:00:18.4482190Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"ce6bad2d-f895-4abf-a47d-456f2916f7a9","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.393,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.3587,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T21:59:20.1274262Z","cpu":3.984,"memory":5191744.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1061,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:59:30.1321959Z","cpu":2.184,"memory":5187212.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0989,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:59:40.1375383Z","cpu":12.266,"memory":5219916.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2751,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T21:59:50.1452870Z","cpu":2.578,"memory":5238660.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0958,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:00:00.1537023Z","cpu":2.574,"memory":5206068.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0611,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:00:10.1673759Z","cpu":13.183,"memory":5262484.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1131,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.3522,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.3481,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.3271,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.324600000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.502900000000004,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T22:00:18.4639343Z","EndTimeUTC":"2026-04-14T22:00:18.4639343Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T22:00:18.4794643Z","DurationInMs":15.530000000000001,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"2a8f8799-4579-4c91-987c-b2e867a8d66a","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26150851,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26150851,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26150851,"SessionToken":"-1#26150851","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.646","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 10:00:18 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T22:00:18.4639343Z", "durationInMs": 0.0082},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T22:00:18.4639425Z", "durationInMs": 19.1104},{"event": "Pipelined", "startTimeUtc": "2026-04-14T22:00:18.4830529Z", "durationInMs": 0.2102},{"event": "Transit Time", "startTimeUtc": "2026-04-14T22:00:18.4832631Z", "durationInMs": 1.8464},{"event": "Received", "startTimeUtc": "2026-04-14T22:00:18.4851095Z", "durationInMs": 0.1282},{"event": "Completed", "startTimeUtc": "2026-04-14T22:00:18.4852377Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T22:00:18.4794643Z","lastSend":"2026-04-14T22:00:18.4794643Z","lastReceive":"2026-04-14T22:00:18.4794643Z"},"requestSizeInBytes":1633,"requestBodySizeInBytes":1181,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T22:00:18.4482190Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"cd6d3657-f62a-4483-95b0-4ea01b1243cd","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T22:00:18.4482190Z","DurationInMs":15.715300000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"235e11f4-26b8-4cb5-aa5a-a97515aadcea","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T22:00:18.4639343Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"2a8f8799-4579-4c91-987c-b2e867a8d66a","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.24999919999999998,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"2a8f8799-4579-4c91-987c-b2e867a8d66a","ResponseTimeUtc":"2026-04-14T22:00:18.4794643Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 2a8f8799-4579-4c91-987c-b2e867a8d66a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26150851","BELatencyInMs":"0.646"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.002}]}
2026-04-14 22:11:48 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 8ab696e7-41ec-499c-ac48-231b54b173c5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 8ab696e7-41ec-499c-ac48-231b54b173c5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 8ab696e7-41ec-499c-ac48-231b54b173c5; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 8ab696e7-41ec-499c-ac48-231b54b173c5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T22:11:48.949Z","duration in milliseconds":42.842600000000004,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T22:11:48.6053878Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95262,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0361},{"name":"Get PkValue From Stream","duration in milliseconds":6.371,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0011},{"name":"Read Collection","duration in milliseconds":6.2091,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T22:11:48.9491453Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"d824a7de-89ff-4961-8334-da9ebdd668f5","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":36.2138,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":36.174600000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T22:10:50.7620946Z","cpu":1.406,"memory":5138092.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1049,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:11:00.7664640Z","cpu":3.906,"memory":5151404.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.141,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:11:10.7821886Z","cpu":7.176,"memory":5148596.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1503,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:11:20.7858421Z","cpu":1.560,"memory":5152480.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1222,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:11:30.7943877Z","cpu":2.266,"memory":5156552.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2705,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1},{"dateUtc":"2026-04-14T22:11:40.7985767Z","cpu":11.719,"memory":5162056.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0373,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":5}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":36.1655,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":36.161,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":36.1336,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":36.130700000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":34.8795,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T22:11:48.9647824Z","EndTimeUTC":"2026-04-14T22:11:48.9647824Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T22:11:48.9960218Z","DurationInMs":31.2394,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"8ab696e7-41ec-499c-ac48-231b54b173c5","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26151994,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26151994,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26151994,"SessionToken":"-1#26151994","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.721","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 10:11:48 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T22:11:48.9647824Z", "durationInMs": 0.0084},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T22:11:48.9647908Z", "durationInMs": 15.2446},{"event": "Pipelined", "startTimeUtc": "2026-04-14T22:11:48.9800354Z", "durationInMs": 0.229},{"event": "Transit Time", "startTimeUtc": "2026-04-14T22:11:48.9802644Z", "durationInMs": 2.0845},{"event": "Received", "startTimeUtc": "2026-04-14T22:11:48.9823489Z", "durationInMs": 0.1376},{"event": "Completed", "startTimeUtc": "2026-04-14T22:11:48.9824865Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T22:11:48.9805005Z","lastSend":"2026-04-14T22:11:48.9805005Z","lastReceive":"2026-04-14T22:11:48.9805005Z"},"requestSizeInBytes":1609,"requestBodySizeInBytes":1157,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T22:11:48.9491453Z","DurationInMs":15.6371,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"276210c0-9de5-40b7-b7da-2481b7e7082e","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T22:11:48.9647824Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"1b4a4780-3ace-4c3c-8efd-14a5834c8cf1","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T22:11:48.9647824Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"8ab696e7-41ec-499c-ac48-231b54b173c5","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.390634,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"8ab696e7-41ec-499c-ac48-231b54b173c5","ResponseTimeUtc":"2026-04-14T22:11:48.9960218Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 8ab696e7-41ec-499c-ac48-231b54b173c5, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26151994","BELatencyInMs":"0.721"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0019}]}
2026-04-14 22:12:20 | Log4Net | Unhandled exception in DiscardArchivalMaterials: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6fa491da-69a6-4137-aefa-786859a2a403; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6fa491da-69a6-4137-aefa-786859a2a403, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 6fa491da-69a6-4137-aefa-786859a2a403; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 6fa491da-69a6-4137-aefa-786859a2a403, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T22:12:20.245Z","duration in milliseconds":4.8289,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T22:12:19.4173652Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95272,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0517},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":4.581,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":4.5385,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T22:11:20.7858421Z","cpu":1.560,"memory":5152480.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1222,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:11:30.7943877Z","cpu":2.266,"memory":5156552.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2705,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":1},{"dateUtc":"2026-04-14T22:11:40.7985767Z","cpu":11.719,"memory":5162056.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0373,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":5},{"dateUtc":"2026-04-14T22:11:50.8050513Z","cpu":22.543,"memory":5104580.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0813,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:12:00.8058495Z","cpu":3.203,"memory":5102852.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.107,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:12:10.8098496Z","cpu":3.750,"memory":5100064.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1076,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":4.5298,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":4.5255,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":4.4985,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":4.4957,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":2.8287,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T22:12:20.2455052Z","DurationInMs":0,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"6fa491da-69a6-4137-aefa-786859a2a403","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26151998,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26151998,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26151998,"SessionToken":"-1#26151998","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.508","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 10:12:20 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T22:12:20.2455052Z", "durationInMs": 0.0039},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T22:12:20.2455091Z", "durationInMs": 0.0044},{"event": "Pipelined", "startTimeUtc": "2026-04-14T22:12:20.2455135Z", "durationInMs": 0.1946},{"event": "Transit Time", "startTimeUtc": "2026-04-14T22:12:20.2457081Z", "durationInMs": 1.8339},{"event": "Received", "startTimeUtc": "2026-04-14T22:12:20.2475420Z", "durationInMs": 0.0742},{"event": "Completed", "startTimeUtc": "2026-04-14T22:12:20.2476162Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T22:12:20.2299582Z","lastSend":"2026-04-14T22:12:20.2299582Z","lastReceive":"2026-04-14T22:12:20.2455052Z"},"requestSizeInBytes":1569,"requestBodySizeInBytes":1119,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.82814,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"6fa491da-69a6-4137-aefa-786859a2a403","ResponseTimeUtc":"2026-04-14T22:12:20.2611297Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 6fa491da-69a6-4137-aefa-786859a2a403, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26151998","BELatencyInMs":"0.508"}}}]}]}]}]}]}]}]}
2026-04-14 22:16:23 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 27017d73-ffc8-47a0-a287-56e37948996d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 27017d73-ffc8-47a0-a287-56e37948996d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 27017d73-ffc8-47a0-a287-56e37948996d; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 27017d73-ffc8-47a0-a287-56e37948996d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T22:16:23.066Z","duration in milliseconds":44.982600000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T22:16:22.3636748Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95309,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.061000000000000006},{"name":"Get PkValue From Stream","duration in milliseconds":7.0608,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0017000000000000001},{"name":"Read Collection","duration in milliseconds":6.8772,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T22:16:23.0668924Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"1d839423-f2d2-422e-8dc9-461b7a415a6f","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":37.6563,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":37.6145,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T22:15:30.9808990Z","cpu":1.794,"memory":5216264.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1008,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:15:40.9850574Z","cpu":3.359,"memory":5210204.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2338,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:15:50.9956832Z","cpu":1.560,"memory":5210728.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0832,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:16:01.0113447Z","cpu":5.616,"memory":5186272.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0471,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:16:11.0135400Z","cpu":1.953,"memory":5261928.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0843,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:16:21.0200019Z","cpu":1.328,"memory":5266196.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2525,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":37.606300000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":37.602900000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":37.58,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":37.5777,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":36.49,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T22:16:23.0824428Z","EndTimeUTC":"2026-04-14T22:16:23.0824428Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T22:16:23.0980640Z","DurationInMs":15.6212,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"27017d73-ffc8-47a0-a287-56e37948996d","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26152091,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26152091,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26152091,"SessionToken":"-1#26152091","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.755","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 10:16:23 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T22:16:23.0824428Z", "durationInMs": 0.0095},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T22:16:23.0824523Z", "durationInMs": 19.0847},{"event": "Pipelined", "startTimeUtc": "2026-04-14T22:16:23.1015370Z", "durationInMs": 0.2035},{"event": "Transit Time", "startTimeUtc": "2026-04-14T22:16:23.1017405Z", "durationInMs": 2.113},{"event": "Received", "startTimeUtc": "2026-04-14T22:16:23.1038535Z", "durationInMs": 0.1571},{"event": "Completed", "startTimeUtc": "2026-04-14T22:16:23.1040106Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T22:16:23.0980640Z","lastSend":"2026-04-14T22:16:23.0980640Z","lastReceive":"2026-04-14T22:16:23.0980640Z"},"requestSizeInBytes":1604,"requestBodySizeInBytes":1156,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T22:16:23.0668924Z","DurationInMs":15.550400000000002,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"549b69c2-7539-4901-a94a-310edb3f3f96","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T22:16:23.0824428Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"8a449954-8a3a-44e4-b0db-858277eb2d50","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T22:16:23.0824428Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"27017d73-ffc8-47a0-a287-56e37948996d","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.7500117,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"27017d73-ffc8-47a0-a287-56e37948996d","ResponseTimeUtc":"2026-04-14T22:16:23.1136865Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 27017d73-ffc8-47a0-a287-56e37948996d, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26152091","BELatencyInMs":"0.755"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0026000000000000003}]}
2026-04-14 22:16:44 | Log4Net | Unhandled exception in DiscardArchivalMaterials: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 961114d4-047b-459f-a051-51980075603a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 961114d4-047b-459f-a051-51980075603a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: 961114d4-047b-459f-a051-51980075603a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: 961114d4-047b-459f-a051-51980075603a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T22:16:44.543Z","duration in milliseconds":4.1605,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T22:16:43.7000545Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95311,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.050100000000000006},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":3.9486000000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":3.9026,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T22:15:50.9956832Z","cpu":1.560,"memory":5210728.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0832,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:16:01.0113447Z","cpu":5.616,"memory":5186272.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0471,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:16:11.0135400Z","cpu":1.953,"memory":5261928.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0843,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:16:21.0200019Z","cpu":1.328,"memory":5266196.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2525,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:16:31.0329703Z","cpu":2.496,"memory":5261188.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.132,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:16:41.0388897Z","cpu":1.328,"memory":5261564.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1937,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":3.8968000000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":3.8927,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":3.8728000000000002,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":3.8693000000000004,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":2.8931,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T22:16:44.5438959Z","DurationInMs":0,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"961114d4-047b-459f-a051-51980075603a","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26152092,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26152092,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26152092,"SessionToken":"-1#26152092","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.579","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 10:16:44 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T22:16:44.5438959Z", "durationInMs": 0.0048},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T22:16:44.5439007Z", "durationInMs": 0.0045},{"event": "Pipelined", "startTimeUtc": "2026-04-14T22:16:44.5439052Z", "durationInMs": 0.19},{"event": "Transit Time", "startTimeUtc": "2026-04-14T22:16:44.5440952Z", "durationInMs": 1.8641},{"event": "Received", "startTimeUtc": "2026-04-14T22:16:44.5459593Z", "durationInMs": 0.0721},{"event": "Completed", "startTimeUtc": "2026-04-14T22:16:44.5460314Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T22:16:44.5282019Z","lastSend":"2026-04-14T22:16:44.5282019Z","lastReceive":"2026-04-14T22:16:44.5438959Z"},"requestSizeInBytes":1571,"requestBodySizeInBytes":1119,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.8438414,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"961114d4-047b-459f-a051-51980075603a","ResponseTimeUtc":"2026-04-14T22:16:44.5438959Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: 961114d4-047b-459f-a051-51980075603a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26152092","BELatencyInMs":"0.579"}}}]}]}]}]}]}]}]}
2026-04-14 22:17:19 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c3f1e009-9c08-480a-823b-cf3aa3cf1a20; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c3f1e009-9c08-480a-823b-cf3aa3cf1a20, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c3f1e009-9c08-480a-823b-cf3aa3cf1a20; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c3f1e009-9c08-480a-823b-cf3aa3cf1a20, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T22:17:19.029Z","duration in milliseconds":45.192800000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T22:17:18.8731360Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95317,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0584},{"name":"Get PkValue From Stream","duration in milliseconds":6.6821,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0042000000000000006},{"name":"Read Collection","duration in milliseconds":6.4929000000000006,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T22:17:19.0294686Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"f17e8939-3cfd-42a4-8e98-a3dbb8210805","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":38.2628,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":38.2192,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T22:16:21.0200019Z","cpu":1.328,"memory":5266196.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2525,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:16:31.0329703Z","cpu":2.496,"memory":5261188.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.132,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:16:41.0388897Z","cpu":1.328,"memory":5261564.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1937,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:16:51.0553699Z","cpu":2.964,"memory":5261372.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0728,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:17:01.0635798Z","cpu":1.404,"memory":5264360.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0939,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:17:11.0711100Z","cpu":2.188,"memory":5255736.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.2576,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":38.211600000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":38.2079,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":38.1841,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":38.1802,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":37.2115,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T22:17:19.0450241Z","EndTimeUTC":"2026-04-14T22:17:19.0450241Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T22:17:19.0762648Z","DurationInMs":31.2407,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c3f1e009-9c08-480a-823b-cf3aa3cf1a20","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26152093,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26152093,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26152093,"SessionToken":"-1#26152093","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.697","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 10:17:19 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T22:17:19.0450241Z", "durationInMs": 0.0084},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T22:17:19.0450325Z", "durationInMs": 19.9686},{"event": "Pipelined", "startTimeUtc": "2026-04-14T22:17:19.0650011Z", "durationInMs": 0.1988},{"event": "Transit Time", "startTimeUtc": "2026-04-14T22:17:19.0651999Z", "durationInMs": 1.9144},{"event": "Received", "startTimeUtc": "2026-04-14T22:17:19.0671143Z", "durationInMs": 0.1594},{"event": "Completed", "startTimeUtc": "2026-04-14T22:17:19.0672737Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T22:17:19.0762648Z","lastSend":"2026-04-14T22:17:19.0762648Z","lastReceive":"2026-04-14T22:17:19.0762648Z"},"requestSizeInBytes":1608,"requestBodySizeInBytes":1156,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T22:17:19.0294686Z","DurationInMs":15.5555,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"098deccd-a4b7-4f2d-bd03-3140853a029f","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T22:17:19.0450241Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"c42c70ab-5e6c-4b81-9996-d43e83ba7fd4","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T22:17:19.0450241Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"c3f1e009-9c08-480a-823b-cf3aa3cf1a20","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.2031288,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c3f1e009-9c08-480a-823b-cf3aa3cf1a20","ResponseTimeUtc":"2026-04-14T22:17:19.0762648Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c3f1e009-9c08-480a-823b-cf3aa3cf1a20, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26152093","BELatencyInMs":"0.697"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0021000000000000003}]}
2026-04-14 22:51:02 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c88c8526-a3de-4a6c-b54b-420356652687; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c88c8526-a3de-4a6c-b54b-420356652687, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: c88c8526-a3de-4a6c-b54b-420356652687; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: c88c8526-a3de-4a6c-b54b-420356652687, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T22:51:02.766Z","duration in milliseconds":22.6771,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T22:51:02.1575183Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95554,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0506},{"name":"Get PkValue From Stream","duration in milliseconds":0.1625,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0015}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":22.290000000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":22.258300000000002,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T22:50:02.7994247Z","cpu":5.382,"memory":5350836.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0852,"availableThreads":8189,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:50:12.8125178Z","cpu":4.609,"memory":5348292.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0688,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:50:22.8202713Z","cpu":4.836,"memory":5230472.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.133,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:50:32.8350311Z","cpu":2.496,"memory":5205508.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0908,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:50:42.8473504Z","cpu":1.950,"memory":5204576.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0991,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T22:50:52.8501201Z","cpu":1.719,"memory":5203980.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0602,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":22.2503,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":22.245800000000003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":22.2283,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":22.2254,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":21.2363,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T22:51:02.7981485Z","DurationInMs":31.244400000000002,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"c88c8526-a3de-4a6c-b54b-420356652687","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26152436,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26152436,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26152436,"SessionToken":"-1#26152436","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.871","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 10:51:02 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T22:51:02.7669041Z", "durationInMs": 0.0158},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T22:51:02.7669199Z", "durationInMs": 17.8928},{"event": "Pipelined", "startTimeUtc": "2026-04-14T22:51:02.7848127Z", "durationInMs": 0.196},{"event": "Transit Time", "startTimeUtc": "2026-04-14T22:51:02.7850087Z", "durationInMs": 2.2306},{"event": "Received", "startTimeUtc": "2026-04-14T22:51:02.7872393Z", "durationInMs": 0.1348},{"event": "Completed", "startTimeUtc": "2026-04-14T22:51:02.7873741Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T22:51:02.7825254Z","lastSend":"2026-04-14T22:51:02.7825254Z","lastReceive":"2026-04-14T22:51:02.7981485Z"},"requestSizeInBytes":1608,"requestBodySizeInBytes":1156,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.64063019999999993,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"c88c8526-a3de-4a6c-b54b-420356652687","ResponseTimeUtc":"2026-04-14T22:51:02.7981485Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: c88c8526-a3de-4a6c-b54b-420356652687, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26152436","BELatencyInMs":"0.871"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0015}]}
2026-04-14 23:11:31 | Log4Net | Error during material processing: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: a4b2ad96-82db-44ed-81ef-8e5fe9961e4a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: a4b2ad96-82db-44ed-81ef-8e5fe9961e4a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: a4b2ad96-82db-44ed-81ef-8e5fe9961e4a; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: a4b2ad96-82db-44ed-81ef-8e5fe9961e4a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T23:11:31.162Z","duration in milliseconds":4.4251000000000005,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T23:11:30.8650484Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95667,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.051500000000000004},{"name":"Get PkValue From Stream","duration in milliseconds":0.16590000000000002,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0018000000000000002}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":4.0467,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":4.0215000000000005,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T23:10:33.9101120Z","cpu":2.734,"memory":5247308.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1039,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T23:10:43.9184504Z","cpu":1.716,"memory":5246640.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1038,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T23:10:53.9267285Z","cpu":3.047,"memory":5246336.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1098,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T23:11:03.9414471Z","cpu":3.900,"memory":5243428.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1198,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T23:11:13.9491588Z","cpu":3.120,"memory":5242200.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0908,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T23:11:23.9587887Z","cpu":1.250,"memory":5242028.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0973,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":4.0169000000000006,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":4.0139000000000005,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":3.9985000000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":3.9963,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":3.0523000000000002,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T23:11:31.1620455Z","DurationInMs":0,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"a4b2ad96-82db-44ed-81ef-8e5fe9961e4a","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26152473,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26152473,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26152473,"SessionToken":"-1#26152473","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.702","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 11:11:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T23:11:31.1620455Z", "durationInMs": 0.0039},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T23:11:31.1620494Z", "durationInMs": 0.0034},{"event": "Pipelined", "startTimeUtc": "2026-04-14T23:11:31.1620528Z", "durationInMs": 0.1769},{"event": "Transit Time", "startTimeUtc": "2026-04-14T23:11:31.1622297Z", "durationInMs": 2.0677},{"event": "Received", "startTimeUtc": "2026-04-14T23:11:31.1642974Z", "durationInMs": 0.0647},{"event": "Completed", "startTimeUtc": "2026-04-14T23:11:31.1643621Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T23:11:31.1463032Z","lastSend":"2026-04-14T23:11:31.1463032Z","lastReceive":"2026-04-14T23:11:31.1620455Z"},"requestSizeInBytes":1615,"requestBodySizeInBytes":1165,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}]},"AuthProvider LifeSpan InSec":0.2969971,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"a4b2ad96-82db-44ed-81ef-8e5fe9961e4a","ResponseTimeUtc":"2026-04-14T23:11:31.1620455Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: a4b2ad96-82db-44ed-81ef-8e5fe9961e4a, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26152473","BELatencyInMs":"0.702"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.001}]}
2026-04-14 23:17:03 | Log4Net | Error during Cosmos DB operations: Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: cf99b96a-d773-4c93-b779-fa16507fd24f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: cf99b96a-d773-4c93-b779-fa16507fd24f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
 Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 1014; ActivityId: cf99b96a-d773-4c93-b779-fa16507fd24f; Reason: (Message: {"Errors":["Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk"]}
ActivityId: cf99b96a-d773-4c93-b779-fa16507fd24f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5);
   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()
--- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 1014)":1},"GatewayCalls":{"(200, 0)":3,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2026-04-14T23:17:03.266Z","duration in milliseconds":60.7682,"data":{"Client Configuration":{"Client Created Time Utc":"2026-04-14T23:17:03.1730368Z","MachineId":"hashedMachineName:6e58dd45-9e7e-c021-0268-275960ac67c8","NumberOfClientsCreated":95697,"NumberOfActiveClients":0,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.46.1|10|X86|Microsoft Windows 10.0.20348 |.NET Framework 4.8.9315.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"System.Web.AspNetSynchronizationContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.055400000000000005},{"name":"Get PkValue From Stream","duration in milliseconds":6.9811000000000005,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0022},{"name":"Read Collection","duration in milliseconds":6.8193,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T23:17:03.2667885Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/vitallog/colls/materiallog","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"1e819e1d-93db-42ed-ab7b-08fc0fbeda97","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":53.545500000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":53.5056,"data":{"System Info":{"systemHistory":[{"dateUtc":"2026-04-14T23:16:04.2292098Z","cpu":3.125,"memory":5163916.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1042,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T23:16:14.2421683Z","cpu":1.014,"memory":5163612.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0723,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T23:16:24.2540086Z","cpu":1.716,"memory":5159876.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1302,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T23:16:34.2689978Z","cpu":3.432,"memory":5171100.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1198,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T23:16:44.2792216Z","cpu":2.500,"memory":5171116.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0758,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0},{"dateUtc":"2026-04-14T23:16:54.2875715Z","cpu":1.092,"memory":5170992.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1146,"availableThreads":8190,"minThreads":2,"maxThreads":8191},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":53.4987,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":53.493500000000004,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":53.4731,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":53.470200000000006,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":52.5032,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14100/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793028943814s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14175/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634982s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14489/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134066793213634980s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2026-04-14T23:17:03.2980436Z","EndTimeUTC":"2026-04-14T23:17:03.2980436Z","TargetEndpoint":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fFU1kAA%3d%3d%2fcolls%2fFU1kAIgZdQE%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2026-04-14T23:17:03.3292902Z","DurationInMs":31.2466,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://vitallog-eastus.documents.azure.com/","StoreResult":{"ActivityId":"cf99b96a-d773-4c93-b779-fa16507fd24f","StatusCode":"Forbidden","SubStatusCode":"PartitionKeyQuotaOverLimit","LSN":26152485,"PartitionKeyRangeId":"3","GlobalCommittedLSN":26152485,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":26152485,"SessionToken":"-1#26152485","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be221.documents.azure.com:14482/apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/","RequestCharge":1.57,"RetryAfterInMs":null,"BELatencyInMs":"0.782","ReplicaHealthStatuses":["(port: 14482 | status: Connected | lkt: 4/14/2026 11:17:03 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2026-04-14T23:17:03.2980436Z", "durationInMs": 0.0094},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2026-04-14T23:17:03.2980530Z", "durationInMs": 19.8312},{"event": "Pipelined", "startTimeUtc": "2026-04-14T23:17:03.3178842Z", "durationInMs": 0.2035},{"event": "Transit Time", "startTimeUtc": "2026-04-14T23:17:03.3180877Z", "durationInMs": 2.0332},{"event": "Received", "startTimeUtc": "2026-04-14T23:17:03.3201209Z", "durationInMs": 0.1577},{"event": "Completed", "startTimeUtc": "2026-04-14T23:17:03.3202786Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2026-04-14T23:17:03.3138296Z","lastSend":"2026-04-14T23:17:03.3138296Z","lastReceive":"2026-04-14T23:17:03.3292902Z"},"requestSizeInBytes":1607,"requestBodySizeInBytes":1157,"responseMetadataSizeInBytes":196,"responseBodySizeInBytes":114},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2026-04-14T23:17:03.2667885Z","DurationInMs":15.626000000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"e056d61d-7042-4b91-839b-83a53a4bd4a6","StatusCode":"OK"},{"StartTimeUTC":"2026-04-14T23:17:03.2824145Z","DurationInMs":15.629100000000001,"RequestUri":"https://vitallog-eastus.documents.azure.com/dbs/FU1kAA==/colls/FU1kAIgZdQE=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"e30990de-7629-4feb-813e-ffbfa89bcc66","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2026-04-14T23:17:03.2980436Z","DurationInMs":0,"RequestUri":"https://vitallog-eastus.documents.azure.com//addresses/?$resolveFor=dbs%252fFU1kAA%253d%253d%252fcolls%252fFU1kAIgZdQE%253d%252fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=3","ResourceType":"Document","HttpMethod":"GET","ActivityId":"cf99b96a-d773-4c93-b779-fa16507fd24f","StatusCode":"OK"}]},"AuthProvider LifeSpan InSec":0.1562534,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"cf99b96a-d773-4c93-b779-fa16507fd24f","ResponseTimeUtc":"2026-04-14T23:17:03.3292902Z","StatusCode":403,"SubStatusCode":1014,"RequestCharge":1.57,"RequestUri":"dbs/vitallog/colls/materiallog","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Partition key reached maximum size of 20 GB. Learn more: https://aka.ms/CosmosDB/sql/errors/full-pk\"]}\r\nActivityId: cf99b96a-d773-4c93-b779-fa16507fd24f, Request URI: /apps/3701c09c-6f03-424d-a9a1-649f37274dca/services/c9e74ca2-40d6-42d4-ad04-cbb67b28825f/partitions/58eee5ef-8d5c-4707-bc02-b35db64cb7fe/replicas/134074855762842884p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.20348 cosmos-netstandard-sdk/3.37.5\r\n   at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WritePrivateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__6`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ConsistencyWriter.<WriteAsync>d__22.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Microsoft.Azure.Documents.RequestRetryUtility.<ProcessRequestAsync>d__2`2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Documents.StoreClient.<ProcessMessageAsync>d__19.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<ProcessMessageAsync>d__3.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.<SendAsync>d__2.MoveNext()","RequestSessionToken":null,"ResponseSessionToken":"3:-1#26152485","BELatencyInMs":"0.782"}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0022}]}