GetQueuedCompletionStatus: when it's safe to delete the completion key

[Dennis Mikhailitsky]

Hello,

I use IOCP for a socket server. When a new connection is created, the
connection's session is created in memory dynamically and its address
is passed to CreateIoCompletionPort as the completion key when the
connection's socket is associated with a completion port. My question
is - when it is 100% safe to remove the connection's session (address
of which GetQueuedCompletionStatus returns as the completion key when
an operation on the socket completes) from memory? In other words,
what combination of both the return value and the output parameters of
GetQueuedCompletionStatus is a 100% indication of that
GetQueuedCompletionStatus will never return this same completion key
again?

Thanks in advance

[iunknown]

Post by Dennis Mikhailitsky
Hello,
I use IOCP for a socket server. When a new connection is created, the
connection's session is created in memory dynamically and its address
is passed to CreateIoCompletionPort as the completion key when the
connection's socket is associated with a completion port. My question
is - when it is 100% safe to remove the connection's session (address
of which GetQueuedCompletionStatus returns as the completion key when
an operation on the socket completes) from memory? In other words,
what combination of both the return value and the output parameters of
GetQueuedCompletionStatus is a 100% indication of that
GetQueuedCompletionStatus will never return this same completion key
again?
Thanks in advance

Usually, these are two ways to safe release resource associated with
IOCP.

The first is using refcounting. When issuing a WSARecv/WSASend,
increase
refcount. When GetQueuedCompletionStatus return, decrease refcount.
It is safe to release resouece when refcount is zero.

The second is using a hashtable. For example, pass SocketHandle to
CreateIoCompletionPort as the completion key, and push the pair --
(SocketHandle, Session) -- into hashtable. When
GetQueuedCompletionStatus
return, get the session from the hashtable by the SocketHandle.
When it need to release the session, remove the pair from the
hashtable first.
After that, when GetQueuedCompletionStatus return, it cannot get the
session
from the hashtable any more.

[David Schwartz]

Post by Dennis Mikhailitsky
I use IOCP for a socket server. When a new connection is created, the
connection's session is created in memory dynamically and its address
is passed to CreateIoCompletionPort as the completion key when the
connection's socket is associated with a completion port. My question
is - when it is 100% safe to remove the connection's session (address
of which GetQueuedCompletionStatus returns as the completion key when
an operation on the socket completes) from memory? In other words,
what combination of both the return value and the output parameters of
GetQueuedCompletionStatus is a 100% indication of that
GetQueuedCompletionStatus will never return this same completion key
again?

Every operation will, unless canceled, complete exactly once. If that
doesn't answer your question, then I don't understand your question.

DS

[Dennis Mikhailitsky]

Post by David Schwartz
Every operation will, unless canceled, complete exactly once. If that
doesn't answer your question, then I don't understand your question.
DS

Why could the following happen: a socket is closed in abortive manner
by calling closesocket, in about 15 mins the associated overlapped
sturcture is removed from memory, yet at some point after that
GetQueuedCompletionStatus returns non-zero as the return value, size
== 0, and pointer to the overlapped structure that has already been
removed from memory? This happens very rarely, therefore I cannot
explore the nature of the issue as I cannot reproduce it consistently,
and this drives me crazy.

[Dennis Mikhailitsky]

I can restructure my original question a little bit - is there a way
to remove association between a socket and a completion port
completely and forever? In other words, is there a way to do an action
opposite to what CreateIoCompletionPort does - to make
GetQueuedCompletionStatus immediately stop returning a given
completion key once and forever, no matter what?

[m]

no - the only way to remove an IO completion port association is to close
the handle.

if this hapens in your code, it means that your reference counts are wrong
or you are frering the OVERLAPPED somwhere that is not the IO completion
handler

Post by Dennis Mikhailitsky
I can restructure my original question a little bit - is there a way
to remove association between a socket and a completion port
completely and forever? In other words, is there a way to do an action
opposite to what CreateIoCompletionPort does - to make
GetQueuedCompletionStatus immediately stop returning a given
completion key once and forever, no matter what?

[David Schwartz]

Post by Dennis Mikhailitsky
I can restructure my original question a little bit - is there a way
to remove association between a socket and a completion port
completely and forever? In other words, is there a way to do an action
opposite to what CreateIoCompletionPort does - to make
GetQueuedCompletionStatus immediately stop returning a given
completion key once and forever, no matter what?

I don't see how that will help you, since a thread may already be in
the process of returning from GetQueuedCompletionStatus.

DS

[David Schwartz]

Post by Dennis Mikhailitsky

Post by David Schwartz
Every operation will, unless canceled, complete exactly once. If that
doesn't answer your question, then I don't understand your question.

Why could the following happen: a socket is closed in abortive manner
by calling closesocket, in about 15 mins the associated overlapped
sturcture is removed from memory, yet at some point after that
GetQueuedCompletionStatus returns non-zero as the return value, size
== 0, and pointer to the overlapped structure that has already been
removed from memory?

For the reason I stated above -- every operation will, unless
canceled, complete exactly once. Some operation had been started and
hadn't complete yet, so it must complete later. There is no guaranteed
time frame for operation completion.

Post by Dennis Mikhailitsky
This happens very rarely, therefore I cannot
explore the nature of the issue as I cannot reproduce it consistently,
and this drives me crazy.

I cannot imagine why you would want to call 'closesocket' while an
operation is in progress on the socket. I don't know for sure whether
there's any safe way to do so -- it's not immediately obvious to me
that it's impossible, but it seems really difficult.

Consider:

1) A thread calls 'GetQueuedCompletionStatus' and gets a completion
indication for this socket, but it blocks on a page fault.

2) Some other thread closes the socket.

3) Lots of other stuff happens.

4) The thread in step 1 finally pages in the page it was waiting for
and continues, processing the completion indication it already got for
the socket you closed in step 2.

I would recommend not calling 'closesocket' until you are completely
finished using the socket because it seems like an awful lot of extra
work to do it safely. It will cancel all pending operations, they will
complete with an error. Microsoft says:

"Any pending overlapped send and receive operations ( WSASend/
WSASendTo/ WSARecv/ WSARecvFrom with an overlapped socket) issued by
any thread in this process are also canceled. Any event, completion
routine, or completion port action specified for these overlapped
operations is performed. The pending overlapped operations fail with
the error status WSA_OPERATION_ABORTED."

"An application should not assume that any outstanding I/O operations
on a socket will all be guaranteed to completed when closesocket
returns. The closesocket function will initiate cancellation on the
outstanding I/O operations, but that does not mean that an application
will receive I/O completion for these I/O operations by the time the
closesocket function returns. Thus, an application should not cleanup
any resources (WSAOVERLAPPED structures, for example) referenced by
the outstanding I/O requests until the I/O requests are indeed
completed."

DS

[Len Holgate]

Post by Dennis Mikhailitsky
Why could the following happen: a socket is closed in abortive manner
by calling closesocket, in about 15 mins the associated overlapped
sturcture is removed from memory, yet at some point after that
GetQueuedCompletionStatus returns non-zero as the return value, size
== 0, and pointer to the overlapped structure that has already been
removed from memory? This happens very rarely, therefore I cannot
explore the nature of the issue as I cannot reproduce it consistently,
and this drives me crazy.

If you are using reference counts on the overlapped structure then you
have a bug. Each operation completes once. Closing the socket doesnt
change that. The operations will still complete once. Increment the
reference count when you issue the request, decrement it when the
operation completes.

In your example above, why is the overlapped structure removed from
memory 15 mins after the socket is closed, what causes that removal?

Len
http://www.lenholgate.com

[iunknown]

Post by Dennis Mikhailitsky

Post by David Schwartz
Every operation will, unless canceled, complete exactly once. If that
doesn't answer your question, then I don't understand your question.
DS

Why could the following happen: a socket is closed in abortive manner
by calling closesocket, in about 15 mins the associated overlapped
sturcture is removed from memory, yet at some point after that
GetQueuedCompletionStatus returns non-zero as the return value, size
== 0, and pointer to the overlapped structure that has already been
removed from memory? This happens very rarely, therefore I cannot
explore the nature of the issue as I cannot reproduce it consistently,
and this drives me crazy.

Does the closesocket return OK ? Or has any errno ?
Generally, after calling closesocket is OK, the pending IO operations
will
return by GQCS( GetQueuedCompletionStatus ) immediately.

[Dennis Mikhailitsky]

Hi Everyone,

Looks like the problem is gone. I wasnt sure exactly which sends and
receives get queued to the completion port and which dont, therefore I
didnt not use reference counters at all and was relying on the GQCS's
return code and combination of the out parameters when making a
decision to destroy the overlapped structure, which obviously was a
bad idea. Now I have implemented reference counters and the problem
seems to be gone. Thanks a lot to all of you for helping to solve this
problem.

Post by iunknown

Post by Dennis Mikhailitsky

Post by David Schwartz
Every operation will, unless canceled, complete exactly once. If that
doesn't answer your question, then I don't understand your question.
DS

Why could the following happen: a socket is closed in abortive manner
by calling closesocket, in about 15 mins the associated overlapped
sturcture is removed from memory, yet at some point after that
GetQueuedCompletionStatus returns non-zero as the return value, size
== 0, and pointer to the overlapped structure that has already been
removed from memory? This happens very rarely, therefore I cannot
explore the nature of the issue as I cannot reproduce it consistently,
and this drives me crazy.

Does the closesocket return OK ? Or has any errno ?
Generally, after calling closesocket is OK, the pending IO operations
will
return by GQCS( GetQueuedCompletionStatus ) immediately.

[Len Holgate]

I tend to use the reference counted design that 'iuknown' suggests.

If you want to see an example of this then take a look at my server
framework code that's available here: http://www.lenholgate.com/archives/000637.html

Len