> What should not be on your main thread is any long blocking compute

Isn't that contradicting yourself? I'm pretty sure open() can block.

When it comes to expecting a specific duration, getpid() is blocking. If you run getpid() in a tight loop and then have performance issues you can’t reasonably blame the system.

> This isn't a portable program; it's a Linux program

But the interface is a portable interface

> POSIX does not mandate that close blocks on anything other than removing the index from the fd table

And what if the fd-table is a very large hash table with high collision rate? How do you then specify how quickly close() should complete? 1ms/open fd? 10ms/open fd? Etc.

It should be clear that the problem here is that the author of the code had a faulty understanding of the system in which their code runs. Today the issue was close() just happened to be too “slow.” If the amount of input devices were higher, let’s say 2x more, then the same issue would have manifested even if close() were 2x “faster.” No matter how fast you make close() there is a situation in which this issue would manifest itself. I.e. the application has a design flaw.

The correct thing to do when you don't care about the result of close(2) is to call it in a worker thread. Ideally there would be async system calls for everything including closing open resources. Ideally there would be only async system calls except for a "wait for next event on this(these) handle(s)" system call.

The proposed patch isn’t a comprehensive fix, it admits there are still other sources of relatively high close() latency.

So that got me thinking, there is no way to fix this “bug” because there is no specification on how long close() should take to complete. As far as we are promised in user-land, close() is not an instantaneous operation. close() is a blocking operation! Even worse, it’s an IO operation.

So now I think the bug is in the application. If you want to avoid the latency of close() you should do it asynchronously in another thread. This is similar to the rule that you should not do blocking IO on the main thread in an event-loop based application.

What should not be on your main thread is any long blocking compute, which is why rendering/game logic often goes to another thread - although simple games could easily be single-threaded.

Good luck. Splice from file to socket doesn’t send any data — it just references it. So if you mmap a file writably and then splice from it to a pipe, either splice needs to copy it or write-protect the mapping to enable copy-on-write or take a reference that will potentially propagate future changes from the mapping to the pipe. Write-protecting the mapping is slow, so pretty much all options don’t work. Blocking until the reference is gone makes no sense, because reference won’t go away after until the caller splices to the network, which will never happen. Deadlock.

So if you're latency sensitive then all of your code needs to be aggressive at avoiding object creation. All of your code becomes part of "the fast path", even if it's in a different thread.

Or you isolate your fast path in a different process or a non-GC'd runtime, the later being the approach taken here by Instagram.

Blocking simply means that the specification does not guarantee an upper bound on the completion time. There is no other meaningful definition. POSIX is not an RTOS therefore nearly all system calls block. The alternative is that the specification guarantees an upper bound on completion time. In that case what is an acceptable upper bound for close() to complete in? 1ms? 10ms? 100ms? Any answer diminishes the versatility of the POSIX VFS.

> Syscalls should be fast unless there is a very good reason not to be.

I think this is an instance of confusing what should be with what is. We’ve been through this before with O_PONIES. The reality is that system calls aren’t “fast” and they can’t portably or dynamically be guaranteed to be fast. So far the only exception to this is gettimeofday() and friends.

Robust systems aren’t built on undocumented assumptions. Again, POSIX is not an RTOS. Anything you build that assumes a deterministic upper bound to a blocking system call execution time will inevitably break, evidenced by OP.

I had to look this one up for a refresher, but 100% violently agree - Such behavior certainly warrants a bug submission.

By that definition, getpid() is definitely nonblocking, though it doesn't have an upper bound in execution time. POSIX does not offer hard realtime guarantees.

close() in general would probably be blocking (as a filesystem may need to do I/O), but I'd expect it to behave nonblocking in most cases, especially when operating on virtual files opened read-only. Unfortunately, I don't think those kinds of behavioral details are documented.

Which is what the author wants. He wants the socket to close once both generate-data and consume-data are done with the socket.

Generally, in most languages, file handles and sockets are automatically closed, when the variable holding them gets out of scope.

Looks like a trade-off which may favor speed over reliability. If a syscall (like msync) in 2nd process will return an error there will no way to abort a build if 1st (main) process already has finished.

Imagine you wanted to generalise it: We have a goroutine that may use some resources: sockets, files, keyboard, screen etc. To shut it down you cut it off from all those resources and hope that it will die.

It sounds wrong somehow.

EDIT: or maybe just allow CLOSED guards in WITH, similar to SEND or RECEIVE, like

  WITH
    SEND(TxChan, text) DO
    | RECEIVE(FailChan, fail) DO
    | CLOSED(FailChan) DO
  END

EDIT2: yet another idea:

  WITH
    SEND(TxChan, text) DO
    | ok := RECEIVE(FailChan, fail) DO
  END

In the case of a blocked process, you may be able to force kill it at a system level but then you risk uncleaned up state (leaked connections/resources)

For instance, this is the default behavior with Postgres (queries complete even if the connection is closed)

Or, busy wait in a hardware state using MONITOR / MWAIT

Just call close immediately.

In fact, there is almost no need for anyone ever to call shutdown.

It may make sense if you want to pass a socket descriptor to another process and you only want them to read from the socket, not write to it, to call a unidirectional shutdown on it before passing it on. But shutdown RDWR and then close is just silly.

The proper fix is not to add more exception handling but to remove the shutdown line altogether.

Shift + F10 > taskmgr > Network Connection flow > End task