/*++ Copyright (c) 2000 Microsoft Corporation Module Name: GC.cxx Abstract: The garbage collection mechanism common code. See the comment below for more details. Author: Kamen Moutafov (kamenm) Apr 2000 Garbage Collection Mechanism: This comment describes how the garbage collection mechanism works. The code itself is spread in variety in places. Purpose: There are two types of garbage collection we perform - periodic and one-time. Periodic may be needed by the Osf idle connection cleanup mechanism, which tries to cleanup unused osf connections if the app explicitly asked for it via RpcMgmtEnableIdleCleanup. The one-time cleanup is used by the lingering associations. If an association is lingered, it will request cleanup to be performed after a certain period of time. The garbage collection needs to support both of those mechanisms. Design Goals: Have minimal memory and CPU consumption requirements Don't cause periodic background activity if there is no garbage collection to be performed. Guarantee that garbage collection will be performed in a reasonable amount of time after its request time (i.e. 10 minutes to an hour at worst case) Implementation: We use the worker threads in the thread pools to perform garbage collection. There are several thread pools - the Ioc thread pool (remote threads) as well as one thread pool for each LRPC address. Within each pool, from a gc perspective, we differentiate between two types of threads - threads on a short wait and threads on a long wait. Threads on a short wait are either threads waiting for something to happen with a timeout of gThreadTimeout or less, or threads performing a work item (threads doing both are also considered to be on a short wait). Threads on a long wait are threads waiting for more than that. As part of our thread management we will keep count of how many threads are on a short wait and how many are on a long wait. All threads in all thread pools will attempt to do garbage collection when they timeout waiting for something to happen. Since all thread pools need at least one listening thread, all thread pools are guaranteed to have a thread timing out once every so often. The garbage collection attempt will be cut very short if there is nothing to garbage collect, so the attempt is not performance expensive in the common case. The function to attempt garbage collection is PerformGarbageCollection If a thread times out on the completion port/LPC port, it will do garbage collection, and then will check whether there are more items to garbage collect (either one-time or periodic) and how many threads from this thread pool are on a short wait. If there is garbage collection to be done, and there are no other threads on short wait, this thread will not go on a long wait, but it will repeat its short wait. This ensures timely garbage collection. If all the threads have gone on a long wait, and a piece of code needs garbage collection, it will request the garbage collection and it will tickle a worker thread. The tickling consist of posting an empty message to the completion port or LPC port. All the synchronization b/n worker threads and threads requesting garbage collection is done using interlocks, to avoid perf hit. This introduces a couple of benign races through the code, which may prevent a thread from going on a long wait once, but that's ok. In order to ensure that we do gc only when needed, in most cases we refcount the number of items that need garbage collection. --*/ #include #include #include #include #include #include #include #include #include #include #include // used by periodic cleanup only - the period // on which to do cleanup. This is in seconds unsigned long WaitToGarbageCollectDelay = 0; // The number of items on which garbage collection // is needed. If 0, no periodic garbage collection // is necessary. Each item that needs garbage collection // will InterlockIncrement this when it is created, // and will InterlockDecrement this when it is destroyed long PeriodicGarbageCollectItems = 0; // set non-zero when we need to cleanup idle LRPC_SCONTEXTs unsigned int fEnableIdleLrpcSContextsCleanup = 0; // set to non-zero when we enable garbage collection cleanup. This either // happens when the user calls it explicitly with // RpcMgmtEnableIdleCleanup or implicitly if we gather too many // connection in an association unsigned int fEnableIdleConnectionCleanup = 0; unsigned int IocThreadStarted = 0; // used by one-time garbage collection items only! long GarbageCollectionRequested = 0; // The semantics of this variable should be // interpreted as follows - don't bother to cleanup // before this time stamp - you won't find anything. // This means that after this interval, there may be // many items to cleanup later on - it just says the // first is at this time. // The timestamp is in millseconds. DWORD NextOneTimeCleanup = 0; const int MaxPeriodsWithoutGC = 100; BOOL GarbageCollectionNeeded ( IN BOOL fOneTimeCleanup, IN unsigned long GarbageCollectInterval ) /*++ Routine Description: A routine used by code throughout RPC to arrange for garbage collection to be performed. Currently, there are two types of garbage collecting - idle Osf connections and lingering associations. Parameters: fOneTimeCleanup - if non-zero, this is a one time cleanup and GarbageCollectInterval is interpreted as the minimum time after which we want garbage collection performed. Note that the garbage collection code can kick off earlier than that. Appropriate arrangements must be made to protect items not due for garbage collection. If 0, this is a periodic cleanup, and GarbageCollectInterval is interpreted as the period for which we wait before making the next garbage collection pass. Note that for the periodic cleanup, this is a hint that can be ignored - don't count on it. The time is in milliseconds. Return Value: non-zero - garbage collection is available and will be done FALSE - garbage collection is not available --*/ { RPC_STATUS RpcStatus = RPC_S_OK; THREAD * Thread; DWORD LocalTickCount; LOADABLE_TRANSPORT *LoadableTransport; LOADABLE_TRANSPORT *FirstTransport = NULL; DictionaryCursor cursor; BOOL fRetVal = FALSE; LRPC_ADDRESS *CurrentAddress; LRPC_ADDRESS *LrpcAddressToTickle = NULL; if (fOneTimeCleanup) { LocalTickCount = GetTickCount(); // N.B. There is a race here where two threads can set this - // the race is benign - the second thread will win and write // its time, which by virtue of the small race window will // be shortly after the first thread if (!GarbageCollectionRequested) { NextOneTimeCleanup = LocalTickCount + GarbageCollectInterval; GarbageCollectionRequested = 1; #if defined (RPC_GC_AUDIT) DbgPrintEx(77, DPFLTR_WARNING_LEVEL, "%d (0x%X) GC requested - tick count %d\n", GetCurrentProcessId(), GetCurrentProcessId(), LocalTickCount); #endif } } else { // WaitToGarbageCollectDelay is a global variable - avoid sloshing if (WaitToGarbageCollectDelay == 0) WaitToGarbageCollectDelay = GarbageCollectInterval; InterlockedIncrement(&PeriodicGarbageCollectItems); } // is the completion port started? If yes, we will use it as the // preferred method of garbage collection if (IocThreadStarted) { // if we use the completion port, we either need a thread on a // short wait (i.e. it will perform garbage collection soon // anyway), or we need to tickle a thread on a long wait. We know // that one of these will be true, because we always keep // listening threads on the completion port - the only // question is whether it is on a long or short wait thread that // we have. // this dictionary is guaranteed to never grow beyond the initial // dictionary size and elements from it are never deleted - therefore, // it is safe to iterate it without holding a mutex - we may miss // an element if it was just being added, but that's ok. The important // thing is that we can't fault LoadedLoadableTransports->Reset(cursor); while ((LoadableTransport = LoadedLoadableTransports->Next(cursor)) != 0) { if (LoadableTransport->GetThreadsDoingShortWait() > 0) { #if defined (RPC_GC_AUDIT) DbgPrintEx(77, DPFLTR_WARNING_LEVEL, "%d (0x%X) Thread %X: there are Ioc threads on short wait - don't tickle\n", GetCurrentProcessId(), GetCurrentProcessId(), GetCurrentThreadId()); #endif // there is a transport with threads on short wait // garbage collection will be performed soon even without // our help - we can bail out FirstTransport = NULL; fRetVal = TRUE; break; } if (FirstTransport == NULL) FirstTransport = LoadableTransport; } } else if (LrpcAddressList && (((RTL_CRITICAL_SECTION *)(NtCurrentPeb()->LoaderLock))->OwningThread != NtCurrentTeb()->ClientId.UniqueThread)) { LrpcMutexRequest(); // else, if there are Lrpc Addresses, check whether they are doing short wait // and can gc for us CurrentAddress = LrpcAddressList; while (CurrentAddress) { // can this address gc for us? if (CurrentAddress->GetNumberOfThreadsDoingShortWait() > 0) { #if defined (RPC_GC_AUDIT) DbgPrintEx(77, DPFLTR_WARNING_LEVEL, "%d (0x%X) Thread %X: there are threads on short wait (%d) on address %X - don't tickle\n", GetCurrentProcessId(), GetCurrentProcessId(), GetCurrentThreadId(), CurrentAddress, CurrentAddress->GetNumberOfThreadsDoingShortWait()); #endif LrpcAddressToTickle = NULL; fRetVal = TRUE; break; } if ((LrpcAddressToTickle == NULL) && (CurrentAddress->IsPreparedForLoopbackTickling())) { LrpcAddressToTickle = CurrentAddress; } CurrentAddress = CurrentAddress->GetNextAddress(); } // N.B. It is possible that Osf associations need cleanup, but only LRPC worker // threads are available, and moreover, no LRPC associations were created, which // means none of the Lrpc addresses is prepared for loopback tickling. If this is // the case, choose the first address, and make sure it is prepared for tickling if ((LrpcAddressToTickle == NULL) && (fRetVal == FALSE)) { LrpcAddressToTickle = LrpcAddressList; // prepare the selected address for tickling fRetVal = LrpcAddressToTickle->PrepareForLoopbackTicklingIfNecessary(); if (fRetVal == FALSE) { // if this fails, zero out the address for tickling. This // will cause this function to return failure LrpcAddressToTickle = NULL; } } LrpcMutexClear(); } else if (fEnableIdleConnectionCleanup) { // if fEnableIdleConnectionCleanup is set, we have to create a thread if there is't one yet RpcStatus = CreateGarbageCollectionThread(); if (RpcStatus == RPC_S_OK) { // the thread creation was successful - tell our caller we // will be doing garbage collection fRetVal = TRUE; } } // neither Ioc nor the LRPC thread pools have threads on short wait // We have to tickle somebody - we try the Ioc thread pool first if (FirstTransport) { // we couldn't find any transport with threads on short wait - // tickle a thread from the RPC transport in order to ensure timely // cleanup #if defined (RPC_GC_AUDIT) DbgPrintEx(77, DPFLTR_WARNING_LEVEL, "%d (0x%X) Thread %X: No Ioc threads on short wait found - tickling one\n", GetCurrentProcessId(), GetCurrentProcessId(), GetCurrentThreadId()); #endif RpcStatus = TickleIocThread(); if (RpcStatus == RPC_S_OK) fRetVal = TRUE; } else if (LrpcAddressToTickle) { // try to tickle the LRPC address fRetVal = LrpcAddressToTickle->LoopbackTickle(); } return fRetVal; } RPC_STATUS CreateGarbageCollectionThread ( void ) /*++ Routine Description: Make a best effort to create a garbage collection thread. In this implementation we simply choose to create a completion port thread, as it has many uses. Return Value: RPC_S_OK on success or RPC_S_* on error --*/ { TRANS_INFO *TransInfo; RPC_STATUS RpcStatus; if (IsGarbageCollectionAvailable()) return RPC_S_OK; RpcStatus = LoadableTransportInfo(L"rpcrt4.dll", L"ncacn_ip_tcp", &TransInfo); if (RpcStatus != RPC_S_OK) return RpcStatus; RpcStatus = TransInfo->CreateThread(); return RpcStatus; } RPC_STATUS EnableIdleConnectionCleanup ( void ) /*++ Routine Description: We need to enable idle connection cleanup. Return Value: RPC_S_OK - This value will always be returned. --*/ { fEnableIdleConnectionCleanup = 1; return(RPC_S_OK); } RPC_STATUS EnableIdleLrpcSContextsCleanup ( void ) /*++ Routine Description: We need to enable idle LRPC SContexts cleanup. Return Value: RPC_S_OK - This value will always be returned. --*/ { // this is a global variable - prevent sloshing if (fEnableIdleLrpcSContextsCleanup == 0) fEnableIdleLrpcSContextsCleanup = 1; return(RPC_S_OK); } long GarbageCollectingInProgress = 0; DWORD LastCleanupTime = 0; void PerformGarbageCollection ( void ) /*++ Routine Description: This routine should be called periodically so that each protocol module can perform garbage collection of resources as necessary. --*/ { DWORD LocalTickCount; DWORD Diff; #if defined (RPC_GC_AUDIT) DbgPrintEx(77, DPFLTR_WARNING_LEVEL, "%d (0x%X) Thread %X: trying to garbage collect\n", GetCurrentProcessId(), GetCurrentProcessId(), GetCurrentThreadId()); #endif if (InterlockedIncrement(&GarbageCollectingInProgress) > 1) { // // Don't need more than one thread garbage collecting // #if defined (RPC_GC_AUDIT) DbgPrintEx(77, DPFLTR_WARNING_LEVEL, "%d (0x%X) Thread %X: beaten to GC - returning\n", GetCurrentProcessId(), GetCurrentProcessId(), GetCurrentThreadId()); #endif InterlockedDecrement(&GarbageCollectingInProgress); return; } if ((fEnableIdleConnectionCleanup || fEnableIdleLrpcSContextsCleanup) && PeriodicGarbageCollectItems) { LocalTickCount = GetTickCount(); // make sure we don't cleanup too often - this is unnecessary if (LocalTickCount - LastCleanupTime > WaitToGarbageCollectDelay) { LastCleanupTime = LocalTickCount; #if defined (RPC_GC_AUDIT) DbgPrintEx(77, DPFLTR_WARNING_LEVEL, "%d (0x%X) Thread %X: Doing periodic garbage collection\n", GetCurrentProcessId(), GetCurrentProcessId(), GetCurrentThreadId()); #endif // the periodic cleanup if (fEnableIdleLrpcSContextsCleanup) { GlobalRpcServer->EnumerateAndCallEachAddress(RPC_SERVER::actCleanupIdleSContext, NULL); } if (fEnableIdleConnectionCleanup) { OSF_CCONNECTION::OsfDeleteIdleConnections(); } } else { #if defined (RPC_GC_AUDIT) DbgPrintEx(77, DPFLTR_WARNING_LEVEL, "%d (0x%X) Thread %X: Too soon for periodic gc - skipping (%d, %d)\n", GetCurrentProcessId(), GetCurrentProcessId(), GetCurrentThreadId(), LocalTickCount, LastCleanupTime); #endif } } if (GarbageCollectionRequested) { LocalTickCount = GetTickCount(); Diff = LocalTickCount - NextOneTimeCleanup; if ((int)Diff >= 0) { #if defined (RPC_GC_AUDIT) DbgPrintEx(77, DPFLTR_WARNING_LEVEL, "%d (0x%X) Thread %X: Doing one time gc\n", GetCurrentProcessId(), GetCurrentProcessId(), GetCurrentThreadId()); #endif // assume the garbage collection will succeed. If it doesn't, the // functions called below have the responsibility to re-raise the flag // Note that there is a race condition where they may fail, but when // the flag was down, a thread went on a long wait. This again is ok, // because the current thread will figure out there is more garbage // collection to be done, because the flag is raised, and will do // a short wait. In worst case, the gc may be delayed because this // thread will pick a work item, and won't spawn another thread, // because there is already a thread in the IOCP, which is doing a // long wait. This may delay the gc from short to long wait. This is // Ok as it is in accordance with our design goals. GarbageCollectionRequested = 0; OSF_CASSOCIATION::OsfDeleteLingeringAssociations(); LRPC_CASSOCIATION::LrpcDeleteLingeringAssociations(); } else { #if defined (RPC_GC_AUDIT) DbgPrintEx(77, DPFLTR_WARNING_LEVEL, "%d (0x%X) Thread %X: Too soon for one time gc - skipping (%d)\n", GetCurrentProcessId(), GetCurrentProcessId(), GetCurrentThreadId(), (int)Diff); #endif } } GarbageCollectingInProgress = 0; } BOOL CheckIfGCShouldBeTurnedOn ( IN ULONG DestroyedAssociations, IN const ULONG NumberOfDestroyedAssociationsToSample, IN const long DestroyedAssociationBatchThreshold, IN OUT ULARGE_INTEGER *LastDestroyedAssociationsBatchTimestamp ) /*++ Routine Description: Checks if it makes sense to turn on garbage collection for this process just for the pruposes of having association lingering available. Parameters: DestroyedAssociations - the number of associations destroyed for this process so far (Osf and Lrpc may keep a separate count) NumberOfDestroyedAssociationsToReach - how many associations it takes to destroy for gc to be turned on DestroyedAssociationBatchThreshold - the time interval for which we have to destroy NumberOfDestroyedAssociationsToReach in order for gc to kick in LastDestroyedAssociationsBatchTimestamp - the timestamp when we made the last check Return Value: non-zero - GC should be turned on FALSE - GC is either already on, or should not be turned on --*/ { FILETIME CurrentSystemTimeAsFileTime; ULARGE_INTEGER CurrentSystemTime; BOOL fEnableGarbageCollection; if (IsGarbageCollectionAvailable() || ((DestroyedAssociations % NumberOfDestroyedAssociationsToSample) != 0)) { return FALSE; } fEnableGarbageCollection = FALSE; GetSystemTimeAsFileTime(&CurrentSystemTimeAsFileTime); CurrentSystemTime.LowPart = CurrentSystemTimeAsFileTime.dwLowDateTime; CurrentSystemTime.HighPart = CurrentSystemTimeAsFileTime.dwHighDateTime; if (LastDestroyedAssociationsBatchTimestamp->QuadPart != 0) { #if defined (RPC_GC_AUDIT) ULARGE_INTEGER Temp; Temp.QuadPart = CurrentSystemTime.QuadPart - LastDestroyedAssociationsBatchTimestamp->QuadPart; DbgPrintEx(77, DPFLTR_WARNING_LEVEL, "%d (0x%X) LRPC time stamp diff: %X %X\n", GetCurrentProcessId(), GetCurrentProcessId(), Temp.HighPart, Temp.LowPart); #endif if (CurrentSystemTime.QuadPart - LastDestroyedAssociationsBatchTimestamp->QuadPart <= DestroyedAssociationBatchThreshold) { // we have destroyed plenty (NumberOfDestroyedAssociationsToSample) of // associations for less than DestroyedAssociationBatchThreshold // this process will probably benefit from garbage collection turned on as it // does a lot of binds. Return so to the caller fEnableGarbageCollection = TRUE; } } #if defined (RPC_GC_AUDIT) else { DbgPrintEx(77, DPFLTR_WARNING_LEVEL, "%d (0x%X) Time stamp is 0 - set it\n", GetCurrentProcessId(), GetCurrentProcessId()); } #endif LastDestroyedAssociationsBatchTimestamp->QuadPart = CurrentSystemTime.QuadPart; return fEnableGarbageCollection; }