From: Michael Simmons
To: NTBUGTRAQ@LISTSERV.NTBUGTRAQ.COM
Subject: Major bug in NT's File Caching Algorithm
This bug makes for an easy (internal) denial of
service attack.
I came across this bug while trying to download a large
(167MB) sysdiff difference file to a client from a
PDC while seting up an unattended install. The PDC ran
so slow for other clients that people couldn't login
or access file and print services.
It was not a network bandwidth problem because other network
access (web and email) on the same subnet was not effected.
I can across the same problem while gziping a large local file
on my N4.0sp3 (+all hotfixes) workstation.
Microsoft appears to be aware of the problem but their solution
(in SP3) is to solve the symptoms an not handle the cause.
To read about this problem see the following KB articles.
Q163880 COPY Command Causes File Cache to Grow
Q164260 Compressing and Uncompressing Files Cause File Cache to Grow
Q164261 Ntbackup Causes Cache to Grow During Restore
Q151778 Huge Downlevel Print Job Causes File Cache to Grow
Q150415 FPNW Printing Causes File Cache to Grow
Q149658 TCP/IP Printing Causes File Cache to Grow
All of the above have something along the line of "it does
not use the flag FILE_FLAG_SEQUENTIAL_SCAN".
Well fixing up the relevant program or service to set this
flag may seem like a solution to microsoft but it isn't to me.
There must be hundreds of programs both Microsoft and other that
do not set this flag and hence can cause the problem.
michael@ecel.uwa.edu.au
=-=
From Sakari.Kouti@TIETURI.FI Sat Nov 1 21:56:45 1997
Date: Sat, 1 Nov 1997 15:04:31 +0200
From: Kouti Sakari
To: NTBUGTRAQ@LISTSERV.NTBUGTRAQ.COM
Subject: Re: Major bug in NT's File Caching Algorithm
When MS tried to fix caching in SP3, the following peculiarity has
appeared:
With SP3 the COPY command seems not to use cache, but XCOPY does.
This can be seen when for instance copying a 5 MB folder from a server
to the workstation.
Without SP3, everything works as one might expect. With the first copy
Physical Disk: Disk Bytes/sec and Server: Bytes Total/sec are roughly
the same (measured on the server machine). And with the second copy Disk
Bytes is zero, since the data being read is already in the cache.
But with SP3 the cache stops working. The second copy reads the same
bytes from the server disk as the first one. Even though they should be
in the cache already.
Now when you switch from COPY to XCOPY, the cache is used again, and
Disk Bytes/sec is zero with the second XCOPY.
Yours, Sakari Kouti, MCSE, MCT
PS. The article Q163880 is either withdrawn from www.microsoft.com, or I
can't use the new support interface. I finally did find the link in
Technet pages of www.microsoft.com, but clicking it gave the result "no
articles found".
=-=
From ortmann@INFORMATIK.TU-MUENCHEN.DE Sat Nov 1 21:57:09 1997
Date: Sat, 1 Nov 1997 18:57:29 +0100
From: Mathias Ortmann
To: NTBUGTRAQ@LISTSERV.NTBUGTRAQ.COM
Subject: Re: Major bug in NT's File Caching Algorithm
NT's greedy file cache triggers another, possibly related problem
that I've been trying to get fixed since November '96. Microsoft
managed to reproduce the bug and promised to fix it - nevertheless,
it is still there, even in NT 5.0 beta 1. With both the file cache
misbehaviour and the bug described below fixed, NT could operate
much more smoothly.
Problem description (originally posted on 12/10/96):
A nasty bug in NT 4.0's virtual memory manager causes all available
memory to be irrecoverably lost every time the system fails to grow
the page file (either because it runs out of disk space, or because
the configured maximum size has been reached). Since NT makes quite
extensive use of the page file even when plenty of physical memory
appears to be available (e.g., a growing file cache can cause
regular memory to be paged out), the only plausible way of
minimizing the risk of suffering a memory collapse as described
above is to make sure that the page file is sufficiently large to
cover even the most extreme situations.
Once memory has collapsed, it remains lost until the system is
rebooted. Memory that has been in use during the collapse is
not affected and can be freed and re-allocated subsequently, but
the collapse is likely to reoccur, once again consuming all of the
RAM that has become available in the meantime.
Microsoft has acknowledged this problem, but seems to be having
difficulty providing a fix quickly.
Reproducing the phenomenon is a fairly straightforward procedure:
- Configure a very small page file (e.g. 2 MB initial / 3 MB
maximum if you have 32 megs of physical memory or more)
(Control Panel / System / Performance / Change)
- Reboot, start Windows NT Diagnostics (Administrative Tools)
and click on the "Memory" tab. If you want to see the effect
graphically, start Task Manager as well and select the
"Performance" sheet.
- Now copy at least _half_ the amount of free physical memory
worth of files from one folder to another. This will cause
your file cache to overflow. In consequence, regular memory
will be swapped out. The page file usage will increase
accordingly (you can monitor this by clicking on "Refresh"
in Windows NT Diagnostics repeatedly).
- Each time the page file usage approaches 100%, a
"System process - out of virtual memory" warning will pop up,
after which its size will grow by a couple of hundred
kilobytes.
- As soon as the page file size hits its upper limit, memory
usage will jump to 100% in an instant (look at Task Manager's
Memory Usage History). From there on, your system will not
be quite as usable as before. Reboot to recover the lost memory.
--
Mathias Ortmann ortmann@informatik.tu-muenchen.de
|