gsadmin/pulse
1
0
Fork 0
mirror of https://github.com/rcourtman/Pulse.git synced 2026-10-24 04:33:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
46d2727828d20680cdc6d3a1ff1b419c3ec2bca4
pulse/.github/scripts/internal-api-test-seconds.txt
T
courtmanr@gmail.com 341e8a7eb4 Calibrate internal/api shard weights to CI and record per-test seconds 
The weights from #2324 came from one local run and left the api shards at
4.5, 8.8, 17.1 and 12.5 minutes on CI. Most of the 17.1 was not test time.
That shard ran `-run . -skip <1231 names>`, and the test binary caches only
its last compiled pattern, so alternating the two recompiled the 62 KB skip
regex for every test and subtest (about 50 ms each under -race locally, 30s
against 1.3s for 300 trivial tests). Two runs put that shard at 887s and
875s against roughly 450s of tests. The skip path now passes -skip alone.

Every api shard now runs go test -json through
.github/scripts/record-internal-api-test-seconds.py. It prints what plain go
test would (package lines and the output of failing or unfinished tests),
exits non-zero on any failure behind pipefail, and writes each top-level
test's seconds to an internal-api-test-seconds-N artifact that is uploaded
even when the shard fails. .github/scripts/refresh-internal-api-test-seconds.py
rebuilds the weights file from those artifacts (gh run download, median
across runs, DEFAULT_WEIGHT set to the mean of the unlisted tests, which the
selector now reads from the file).

Until those artifacts exist, the weights are a local -race -json run with
each region of go test's order scaled to what CI reported for it.
Equal-count quarters averaged 107, 437, 100 and 787s over four runs, and the
weighted cut gave 131s for the first 1187 tests, 350s for the next 23 and
608s for the last 21 over two. The last quarter and the last 21 tests
disagree on how the tail splits, so the tail takes the midpoint of an exact
fit and a pooled factor.

With these weights four shards still predict a slowest shard near 10
minutes, and a 20% slower runner on the tail would take it past the 11
minutes of Frontend and rest-1. Five shards predict about 131, 325, 280-340,
330-355 and 330-365s of tests, or 4.5 to 8.5 minutes a job with about 2.3
minutes of setup, so API_SHARD_COUNT goes to 5. Every test still runs
exactly once in contiguous go test order, and all five slices passed locally
from their new starting points. The required Backend tests (api) verdict
keeps its name.
2026-09-29 15:27:08 +01:00

218 lines
11 KiB
Plaintext
Raw Blame History

# Run time in seconds of the slowest internal/api top-level tests under -race
# on a GitHub ubuntu-24.04 runner. Used only by select-internal-api-shard.sh
# to choose where the Backend tests (api-N) shards cut go test's list. Tests
# not named here weigh DEFAULT_WEIGHT, the mean of the tests under 0.5s. A
# missing, stale or renamed entry can make the shards uneven but can never
# drop or repeat a test.
#
# Every api shard uploads an internal-api-test-seconds-N artifact with the
# seconds of each test it ran on the runner, so refresh this file from recent
# passing Build and Test runs with
# python3 .github/scripts/refresh-internal-api-test-seconds.py --run <id> [--run <id>]
# when one api shard runs well over the others.
#
# Seeded 2026-09-29, before any such artifact existed, from a local
# GOMAXPROCS=4 go test -race -json run with each region of go test's order
# scaled so its sum matches the seconds the runner reported for it.
# Equal-count quarters (runs 36557693864, 36560070496, 36567708327,
# 36577491881) averaged 107s, 437s, 100s and 787s. The weighted cut (runs
# 36566872567, 36578019732) averaged 131s for the first 1187 tests, 350s for
# the next 23 and 608s for the last 21. Its 2556-test shard (887s, 875s) is
# left out, because it ran through -run . -skip, which recompiles the skip
# pattern for every test and subtest. The last quarter and the last 21 tests
# disagree on how the tail splits, so the two tail regions take the midpoint
# of an exact fit and one pooled factor.
DEFAULT_WEIGHT 0.05
TestActionRunnerRotationProductionRouterTLSPersistenceRestart 0.82
TestMockActionInboxAndDetailUseCanonicalResponses 2.85
TestMockActionInboxHonorsLimitAndLegacyPendingProjection 2.12
TestMockActionMutationsFailReadOnly 1.16
TestAgentReportHandlersClassifyBodyFailures 1.31
TestStart_MockModeStartsChatServiceWhenPersistedConfigDisabled 1.13
TestRestart_MockModeStartsChatServiceWhenPersistedConfigDisabled 1.13
TestHandleChat_EmitsIdleProgressWhileServiceIsSilent 2.85
TestAISettingsHandler_ProviderSettersConcurrentServiceCreation 9.86
TestHandleSaveGuestNote_VerifySavedFields 0.62
TestHandleClearGuestKnowledge_OversizedBody 0.68
TestHandlePatrolStream_LastEventIDHeaderTakesPrecedenceOverQueryFallback 2.37
TestHandleExecuteStream_LicenseRequired 1.25
TestHandleExecuteStream_PromptRequired 1.26
TestHandleExecuteStream_Success 2.13
TestHandleExecuteStream_EmitsIdleProgressWhileServiceIsSilent 2.06
TestHandleInvestigateAlert_EmitsIdleProgressWhileServiceIsSilent 2.01
TestAISettingsHandler_PatrolReadinessBranches 1.01
TestAISettingsHandler_KeylessOpenAICompatibleSetupIsConfiguredAndValidated 0.51
TestAISettingsHandler_TestProvider_ConnectionFailure 0.52
TestHandleGetAICostSummary_MethodNotAllowed 0.54
TestHandleInvestigateAlert_UsesVisibleIdleProgressStream 2.05
TestHandleGetPatterns_UnlockedWithData 1.23
TestHandleGetPredictions_WithData 1.23
TestHandleGetCorrelations_WithData 1.25
TestHandleGetCorrelations_UsesCanonicalIntelligenceFacade 1.24
TestExtractAndStoreAuthContext_UsesTenantConfigForToken 3.54
TestAuthDenialPruneKeepsTrackedSetBounded 2.28
TestAvailabilityHandlersListReturnsMockTargetsInMockMode 1.22
TestAvailabilityHandlersTestSavedMockTargetUsesSyntheticStatus 1.22
TestChangePasswordRefusesNonAdminSession 0.90
TestChangePasswordAllowsConfiguredAdminSession 1.66
TestChangePasswordBasicAuthPathUnaffected 4.57
TestChangePasswordFixtureOwnsRouterLifecycle 0.91
TestConfigTransferAnonymousAuthenticatedModesDenyBeforeBodyRead 0.55
TestConfigTransferAuthorizedInstanceModesReachHandler 0.77
TestConfigTransferTenantSessionsRequireManagement 0.54
TestConnectionsHandleListIncludesMockAvailabilityTargets 1.23
TestConnectionsHandleListDropsRealSourcesInMockMode 1.30
TestConnectionsLedgerKeepsRealSourcesWhenRealPollingRetained 1.26
TestContract_MockAvailabilityTargetsUseSavedTargetPayloads 1.23
TestContract_MockDiscoveryEndpointsUseCanonicalPayloads 1.15
TestContract_MockChartRoutesUseCanonicalMockUnifiedReadStateForVMwareHosts 324.29
TestContract_PlatformMockToggleRebindsRuntimeConnectionsAndResources 24.19
TestContract_PlatformMockConnectionListsUseSharedFixtureMetadata 1.31
TestContract_RecoveryPointsMockPathReturnsCanonicalProviderBackedFixtures 2.15
TestContract_SecurityStatusPresentationPolicyProEditionCommercialContext 0.67
TestContract_StateAndResourceListShareCanonicalMockResourceContract 6.16
TestContract_AgentCommandConfigGateMirrorsChannelAdmission 1.50
TestContract_HostedDiagnosticsDockerPrepareTokenValidatesOriginBeforeMutation 1.62
TestContract_HostedInstallerOriginsFailClosedAtRouter 5.36
TestContract_RuntimeDisplayServesPresentationValuesWithoutAdmin 1.11
TestSecurityStatusCurrentUserForScopedLocalSession 1.11
TestSecurityStatusOrgBoundSettingsFollowOrgManagementRule 1.11
TestSecurityStatusOrgBoundReportingRespectsAuthorizer 1.10
TestDiscoveryHandlers_MockModeServesFixturesWithoutService 1.50
TestDockerAgentReportSizeContractAcceptsOrdinaryAndLargeFleets 0.73
TestDockerAgentReportSizeContractBoundaries 3.42
TestHostAgentRemovalLifecycleThroughAuthenticatedRouterAndRestart 0.59
TestCollectorUninstallTransactionFailsClosedAndRetriesAfterRestart 2.85
TestHostedTenantAgentInstallCommand_GeneratesOrgBoundTokenAndCommand 4.15
TestGenerateHostedTenantAgentInstallCommandEnforcesHostedOrgBoundary 1.48
TestHostedTenantAgentInstallCommandCarriesCommandPolicyIntent 1.98
TestHostedTenantAgentInstallTokenCannotReportToOtherTenant 0.67
TestIssue1753SameNameProxmoxAgentsAuthenticateAndStayDistinctEndToEnd 1.73
TestSyncReleaseDemoFixtureRuntime_DoesNotPoliceMockFixturesInDevBuilds 0.96
TestMetricsHistoryFallbackSynthesizesUnseededMockWorkloadHistory 1.29
TestMetricsHistoryFallbackSynthesizesMockHostAndNodeDrawerSeries 1.57
TestMetricsHistoryFallbackSupplementsPartialMockStoreSeries 1.32
TestMetricsHistoryFallbackMockDiskSynthesizesSeries 1.30
TestMetricsHistoryFallbackMockUnifiedStorageSynthesizesSeries 1.79
TestMetricsHistoryFallbackMockStorageBypassesSparseStoreSeries 1.94
TestSSOOnlyExplicitAdminReachesEveryAdminGuard 1.11
TestUnrelatedSSOUserStillRefusedWhenLocalAdminConfigured 1.65
TestPlatformAdminRouteRefusesUnrelatedSSOUser 0.66
TestJourneyGeneralAPIBudgetIsolation 1.38
TestTenantRBACProvider_NoConnectionLeak 1.13
TestHandleListPointsAcceptsCanonicalPlatformQuery 1.43
TestHandleListRollupsExposeCanonicalPlatformsPayload 1.46
TestHandleListRollupsMetaReportsNormalizedPagination 2.13
TestHandleListPointsMetaReportsNormalizedPagination 2.15
TestStartPatrolForContext_DoesNotOverwriteOtherTenantPatrolComponents 1.81
TestStartPatrolForContext_RejectsMismatchedAIServiceOrg 0.98
TestRouterMockMode_SeedsTrueNASAndVMwareSupplementalResources 1.24
TestRouterMockMode_SeedsVMwareSupplementalActivity 0.95
TestRouteExecuteStream_Success 0.74
TestRouteExecuteStream_MethodNotAllowed 0.60
TestRouteExecuteStream_WrongScope 0.51
TestRouteExecuteStream_EmptyPrompt 0.65
TestRouteExecuteStream_WhitespaceOnlyPrompt 0.58
TestRouteExecuteStream_InvalidJSON 0.63
TestRouteExecuteStream_AIDisabled 0.61
TestRouteExecuteStream_AutofixLicenseRequired 0.59
TestRouteExecuteStream_RemediationLicenseRequired 0.52
TestRouteExecuteStream_OversizedBody 0.64
TestRouteExecuteStream_UseCaseCaseInsensitive 0.68
TestRouteExecuteStream_SSEHeaders 0.52
TestRouteExecuteStream_CompleteEventBeforeDone 0.60
TestRouteExecuteStream_EmptyBody 0.54
TestRouteExecuteStream_NullPrompt 0.58
TestRouteExecuteStream_ServiceError 0.55
TestRouteExecuteStream_InvalidHistoryRole 0.55
TestRouteExecuteStream_ValidTargetTypes 1.64
TestRouteExecuteStream_EmptyTargetTypeAllowed 1.18
TestRouteExecuteStream_WithTargetContext 0.59
TestRouteExecuteStream_DefaultUseCaseIsChat 0.58
TestRouteExecuteStream_FindingIDPassedThrough 0.58
TestRouteExecute_Success 0.58
TestRouteExecute_MethodNotAllowed 0.59
TestRouteExecute_NoAuth 0.62
TestRouteExecute_WrongScope 0.55
TestRouteExecute_EmptyPrompt 0.58
TestRouteExecute_InvalidJSON 0.59
TestRouteExecute_WithConversationHistory 0.54
TestRouteExecute_WithTargetContext 0.69
TestRouteExecute_DefaultUseCaseIsChat 0.58
TestRouteExecute_AutofixLicenseRequired 0.59
TestRouteExecute_OversizedBody 0.73
TestRouteExecute_EmptyBody 0.52
TestRouteExecute_EmptyJSONObject 0.74
TestRouteExecute_NullPrompt 0.59
TestRouteExecute_ServiceError 0.58
TestRouteExecute_InvalidTargetType 0.59
TestRouteExecute_ValidTargetTypes 1.70
TestRouteExecute_EmptyTargetTypeAllowed 0.59
TestRouteExecute_InvalidHistoryRole 0.58
TestRouteExecute_FindingIDPassedThrough 0.60
TestRouteExecute_ModelFieldPassedThrough 0.58
TestRouteListModels_Success 0.54
TestRouteListModels_MethodNotAllowed 0.52
TestRouteListModels_WrongScope 0.57
TestRouteListModels_ProviderError 0.59
TestRouteListModels_EmptyModelList 0.60
TestRouteListModels_CachedField 0.60
TestRouteListModels_MalformedProviderResponse 0.61
TestRouteListModels_HEADMethod 0.69
TestRouteTestConnection_Success 0.60
TestRouteTestConnection_MethodNotAllowed 0.57
TestRouteTestConnection_NoAuth 0.58
TestRouteTestConnection_WrongScope 0.58
TestRouteTestProvider_Success 0.57
TestRouteTestProvider_UnconfiguredProvider 0.52
TestRouteTestProvider_MethodNotAllowed 0.60
TestRouteTestProvider_NoAuth 0.59
TestRouteTestProvider_MultipleProviders 0.58
TestRouteTestProvider_WrongScope 0.59
TestRouteTestProvider_InvalidProviderNames 0.58
TestRouteTestProvider_PathTraversal 0.67
TestRouter_HandleState_MockIsolation 3.26
TestAITestConnectionRouteWithValidScope 0.72
TestAITestConnectionRouteRejectsWrongScope 0.79
TestAITestProviderRouteRejectsWrongScope 0.71
TestPublicEndpointsBypassAuthInAPIMode 0.59
TestSettingsCapabilitiesMatchRouteEnforcementWithoutRBAC 0.83
TestSettingsCapabilitiesGrantConfiguredAdminWithoutRBAC 0.90
TestTrueNASHandlers_HandleList_ReturnsMockConnectionsInMockMode 1.18
TestFreeBSDOfflineAgentPreflight 1.77
TestBranchcov0723AmExtractFromZip 8.53
TestHandleApplyUpdate_BlocksWhenReadinessBlocked 1.42
TestHandleGetUpdatePlan 1.59
TestHandleGetUpdatePlan_IncludesUpgradeReadiness 1.29
TestHandleGetUpdatePlan_InvalidChannel 1.05
TestHandleGetUpdatePlan_PrepareError 1.38
TestVMwareHandlers_HandleList_ReturnsMockConnectionsInMockMode 1.28
TestAlertsEndpoints 22.93
TestRecoveryRollupsEndpointToleratesMalformedPersistedMetadata 0.53
TestHealthEndpoint 21.24
TestVersionEndpointUsesRepoVersion 21.71
TestAlertAcknowledge_AllowsPrintableAlertIDs 20.98
TestStateEndpointReturnsMockData 23.11
TestRecoveryRollupsEndpointReturnsMockData 23.03
TestProtectedEndpointsRequireAuthentication 20.77
TestAPIOnlyModeRequiresToken 31.38
TestServerInfoEndpointReportsDevelopment 27.15
TestRecoveryPointsEndpointReturnsMockData 31.33
TestServerInfoEndpointMethodNotAllowed 26.85
TestHealthEndpointMethodNotAllowed 27.02
TestHealthEndpointHeadAllowed 26.90
TestConfigNodesUsesMockTopology 25.70
TestMockModeToggleEndpoint 36.62
TestMockModeToggleFromRealDataRebindsPlatformSupplementalProviders 27.70
TestAuthenticatedEndpointsRequireToken 27.94
TestAPITokenQueryAndHeaderAuth 33.28
TestRecoveryEndpointRequiresDirectLoopback 29.26
TestWebSocketSendsInitialState 27.92
TestWebsocketPayloadContractShape 25.38
TestWebsocketPayloadUsesCanonicalStateContract 28.16
TestSessionCookieAllowsAuthenticatedAccess 29.04
TestRevokedAPITokenImmediatelyLosesAccess 35.04
TestLimitedAPITokenCannotCreateBroaderToken 29.25
TestPublicURLDetectionUsesForwardedHeaders 24.93
TestPublicURLDetectionRespectsEnvOverride 27.56
Reference in New Issue View Git Blame Copy Permalink