"ls" in directory outputs "No such file or directory" for some files

I cd to a directory that has a large number of files, 40,948 to be exact. When I issue ls it takes a long time and before the command finally prints the results to the screen it tells me "ls: cannot access : No such file or directory" for a few files.

It's not the same file names every time. When I do ls -l some files have no posix permissions just "???????" preceding the file name.

I tried to chown -R the directory because I wanted to change the group and I get a mixture of "chown: cannot access" or "chown: changing ownership of" + ": No such file or directory" on even more files than when I use ls.

This highlighted for me that I don't even know if a directory in Linux has a table of contents, I thought that maybe the toc was corrupted. But the fact that the results are always different suggests otherwise.

Could this be a "nofile" issue in /etc/security/limits.conf ?