* Use no more than XX GB disk space.
This is an absolute limit and NOT an invitation for BOINC to overuse. If BOINC needs to use 100MB then that's exactly what it will use, irrespective of whether or not you give it permission to use a whole lot more. It is entirely safe to set it to 10GB or even 100GB even if you only have a 2GB disk, as I have in one particular case. BOINC is always limited to the physical size of the partition on which it is installed anyway.
I did try it again and I came up with the same comment :).
Thanks very much for the link. It supports my case very nicely :).
Here is what the user reported:-
Quote:
3/09/2005 5:36:08 a.m.|SETI@home|Message from server: (there was work but you don't have enough disk space allocated)
3/09/2005 5:36:08 a.m.|SETI@home|Message from server: No disk space (YOU must free 17889.5 MB before BOINC gets space). Review preferences for minimum disk free space allowed.
3/09/2005 5:36:08 a.m.|SETI@home|No work from project
C: 40GB - 7.8 Used - 32 Free
D: 35GB - 1.6 GB Used (Temp Filesystem)
e: 160GB - 146.9 Used 13 free
F: 200GB - 130 Used 69 Free.
All are actual hard disks - not shares or partitioned disk.
Suggestions oh wise ones ?
In a followup message the user also mentioned:-
Quote:
Hmmm - had leave 50 GB disk space free - which would have caused problem - usual story cannot see wood for trees.
The user never did mention the actual BOINC partition or what his preferences (apart from the free space) actually were set at. Let's assume BOINC is on C:. Let's also assume he has the first disk pref set at "Use no more than 100GB disk space". We know the second pref was "Leave at least 50GB disk space free" because the user admitted that this was the problem. It also fits nicely with the BOINC error message of needing to "free 17889.5 MB". Here are the maths:-
40Gb - 7.8GB = 32.2GB free space. 50 - 32.2 = 17.8GB which is the shortfall in free space with the unfortunate 50GB setting. As you can see, this has nothing to do with the maximum disk space setting.
Quote:
If not changed and left to its own devices, BOINC will always try to take 100GB. Yes, there are drives bigger than that already. :)
Of course there are. These days the cheapest drives to buy are 200GB. However, I strongly dispute the first part of the quote. I've had machines running for months with disk sizes between 2GB and 200GB and a preference of 10GB. I've just checked the BOINC folder on a machine running EAH and LHC on which BoincView is also installed as a subfolder. The BOINC folder is 65MB. Part of this is BoincView at 34.4MB (logs 33.8MB - ouch!!) so actually BOINC is about 0.03GB. Hey, what's wrong with that stupid SOB!! Doesn't it realise I've given it permission to use 10GB?? Why is it only using 0.03GB :).
Over the last two days, I've looked at quite a few (20 - 100, I've really lost track) reports of disk usage problems, going back to 2004. Many are obviously the result of user error - just like the one you quoted. Many don't have enough data - don't you just hate it when you spend time trying to work it out and then ask for more information, only to be told, "Don't worry, I made a few random changes - don't remember what I did - seems to be sorted now, bye bye!!" :)
I've reached the conclusion that there is some "funny business" with that third preference setting but that the first two are NOT an issue. You can set the first one to whatever you want as long as it is large enough for future expansion and it doesn't matter about the actual partition size. You will, of course, be limited to that. You should set the second one basically to zero since no other software package seems to worry too much about running out of disk space :).
The third one is quite an enigma and the more I look, the more it seems that someone should really look closely at the code where it is used because there may be something funny going on. Unfortunately, that's not going to be me - programmer I ain't :).
Bruce Allen and Walt Gribben have recent posts here that may shed some light on the problems described here and on similar threads. However, since we don't have a whole lot of data on the problem, I am not sure that their admissions clear up everything. I guess time will tell.
I've been away for a couple of days and I check back and it seems like all hell's broken loose. I see that you, Michael, Bill, Walt, Kathryn and others have been doing an amazing job with handling the influx of new users. I've seen the rapid rollout of about 5 new BOINC versions to handle the problems revealed by the upsurge in users particularly over at Seti, and I've seen the comments by Bruce and Walt that you refer to and also the message from Bruce about their awareness of the need to fix 4.79 issues. Hopefully that might go a small way towards answering the valid concerns you raise about having to continually direct people to the beta app.
I'm going to go ask for clarification from Walt about what bug he found and why it seemed like a good idea at the time to equate zero with 100MB rather than zero really being zero. If it really cant be zero then why not make it 0.5K
...I've seen the rapid rollout of about 5 new BOINC versions
Gary,
Add another version to the list - 5.2.13 looks to correct some more of these recently-surfaced issues.
I haven't quite been pulling my weight on the boards recently. My rig's gotten balky with the little warmup in weather here, and I've been loathe to give back any hard-won ground in the overclocking battle. Firefox and Einstein haven't been playing nice together, especially since I've gone to 1.5 release candidates (1, 2, now 3), but I think it's mostly a heat issue, with ambient temps back upto 18C.
(edit) - I'm back now, after a 5-minute breather. I wasn't finished with the post, but the rig has this funny habit of locking or shutting down and rebooting, and I've completely lost a couple of long, hard-to-compose would-be posts due to it's abberant behavior. The good news is that, miraculously, no Einstein work has been lost or corrupted.
microcraft
"The arc of history is long, but it bends toward justice" - MLK
Perhaps a review of how the "available space" is calculated would help.
BOINC sends your preferences and disk usage numbers to the scheduler when it requests work. Thats what actually gets used, you can see them in the sched_request_einstein.phys.uwm.edu.xml file.
The scheduler does some calculations with the numbers and then uses the smallest result.
For "new" clients that report the space BOINC uses, it calculates:
"use no more than __GB" minus "disk space used by BOINC"
("use no more than __%" times "total disk space") minus "disk space used by BOINC"
"disk free space" minus "leave at least __GB free"
and keeps the smallest. (it turns your GB preference into bytes, and your % into a fraction, I just left those out to make it easier to read)
Also note that "leave at least __GB free" can't be smaller than .001 (1,000,000 bytes), if you set something smaller the scheduler uses .001 instead of your preference. That leaves some slack for administrative files (client_state, scheduler requests/replies, std*.txt, stuff like that)
Older BOINC clients don't send the space BOINC uses, so it only makes one calculation:
"host free disk space" minus "leave at least __GB free"
Thats it for determining how much it can use.
Once the "available space" is calculated, it makes sure theres enough space for the workunit.
While its building a list of workunits to send, the scheduler keeps track of the disk space needed by the workunit. It counts:
space needed for each file not already on the host
space needed to run the workunit (100MB or 100,000,000 bytes)
So the minimum space needed is 101 megabytes (101,000,000 bytes) plus the space needed for any files it has to sent. At least for Einstein@Home, other projects have other requirements.
If you check the messages from the scheduler, it'll tell you which preference generated the smallest "available disk space" value, and whether you even have that much available.
The message "Not enough disk space (only __ MB free for BOINC)." means you have some disk space BOINC can use, but its not enough to send a workunit.
The message "No disk space (YOU must free __ MB before BOINC gets space)." means you don't even have the minumum ("available disk space" is negative), thats how much you have to free up just to get to zero.
And remember, you need 100MB more that that, and maybe another 10MB if it has to download new work files. Seti@Home only needs 500,000 bytes "working space" while CPDN needs more. Depends on the workunit.
Some notes on where things come from...
"Disk space" only uses the drive BOINC runs from. That info is written to BOINC's message log when it starts up, along with the total and free disk space. So for hosts with multiple partitions and multiple drives, it ignores all that free space on the "next partition over".
Current clients (BOINC 5.2) send the total disk space, free disk space and the disk space used by BOINC and all projects when it contacts the scheduler. So they're using current numbers - it gets the disk space info from the operating system and runs thru all the files and folder in the BOINC folder adding up whats used. Actual date for the change was October 4, so clients built after that should be sending the proper data.
Some earlier versions didn't always send the correct disk space, it read it at startup and kept sending the same numbers each time it contacted the scheduler. BOINC 4.19 was the last one that tracked disk space correctly that I know of, but it only sent the total disk space and free disk space. So BOINC versions 4.25 thru 5.1 may not be sending the correct space info.
If in doubt, look in the sched_request_einsten.phys.uwm.edu.xml file to see what actually got sent, then compare it to whats actually on your disk. If they're very different, and its stopping you from getting work, then perhaps its a good reason to update your BOINC client.
The tags are:
- total disk space
- free disk space
- total disk space used by BOINC
The preferences used by the scheduler come from your host, BOINC sends them in the scheduler request. So what you see in your preferences on the web site might not be what the scheduler uses. Check BOINC's message log to see what actually got sent and where they came from. Might actually be preferences from another project.
If in doubt, check the file sched_request.einstein.phys.uwm.edu,xml, they're in the " section.
There are times the scheduler uses its own defaults for preferences. Thats when BOINC doesn't have any to send, and there aren't any set for the user on the web site. New users attaching with the "create new account" option in BOINC have that problem, its an oversight thats being corrected.
Which is where the earlier "no disk space" problems came from, the defaults used by the scheduler were too small for the Einstein workunits, unless you had a very large drive.
Its easy to tell if you don't have preferences set, the "Preferences last modified" line near the top won't have a date.
Perhaps a review of how the "available space" is calculated would help.
Walt,
I'm extremely grateful for the time you have spent to document all this. It's a marvellous resource and it will take me a while to digest it all. It might have to wait a bit until after the flood of new users begins to subside, but it's something I really want to understand better than I currently do so I'm really grateful to have the information.
I was very interested to see that versions after 4.19 and before 5.2.x might be sending bad info as that would probably explain a number of puzzling reports that I've seen in the past. Thanks again for what you have done, and particularly for all the marvellous detail :).
Thanks to Gary, Walt, and others for the in-depth answers.
Regarding disk space usage preferences and remaining disk space: When the problem happened to me, I suppose the disk could have been very close to 50% full, and not quite enough left for BOINC and E@H to do another WU.... I can't imagine why I haven't seen the problem resurface since then as I have installed quite a bit of software since then (Microsoft antispyware, Ad Aware, Encyclopedia Britannica, virus and firewall updates, etc.), and haven't done any purposeful disk clean-up.
The disk is very close to 50% full now (just 1GB to go), so I'm going to see if I get the disk space error again. If so, I will come back to this thread with a report of the appropriate .xml files and results of any changes I make.
Very interesting discussion, and I hope I can provide some data.
The disk is very close to 50% full now (just 1GB to go), so I'm going to see if I get the disk space error again. If so, I will come back to this thread with a report of the appropriate .xml files and results of any changes I make.
Thank you for doing that!! A report back would be much appreciated, even if only to clarify the actual values that you can see if the problem happens to strike again.
Just a little update: I got tired of waiting for the "problem" to show up, so I duplicated a little more than a GByte of MP3 files. Two WUs have now downloaded.
sched_request.einstein.phys.uwm.edu.xml does report correctly. So now more than 50% of the disk is used up. No disk space errors yet.
Just a little update: I got tired of waiting for the "problem" to show up, so I duplicated a little more than a GByte of MP3 files. Two WUs have now downloaded.
sched_request.einstein.phys.uwm.edu.xml does report correctly. So now more than 50% of the disk is used up. No disk space errors yet.
Whats using the over 50% of the disk? If its BOINC then theres a problem that has to be fixed. If its something else, thats not a BOINC issue - the preference says how much disk space BOINC (and the BOINC applications) can use.
You can test this yourself by setting the percentage to match what BOINC actually uses on the disk. But don't do this while you're processing workunits, or have workunits queued up. BOINC is likely to abort them with an "exceeded disk space" message. Clear out your work queue (all projects), then try testing. You can also test "leave at least __GB free" by setting the preference to your current free disk space, and check the "Use at most..." by setting the preference to what BOINC actually uses. And remember to set them back to reasonable values when done testing.
RE: RE: The three key
)
Jord,
I did try it again and I came up with the same comment :).
Thanks very much for the link. It supports my case very nicely :).
Here is what the user reported:-
In a followup message the user also mentioned:-
The user never did mention the actual BOINC partition or what his preferences (apart from the free space) actually were set at. Let's assume BOINC is on C:. Let's also assume he has the first disk pref set at "Use no more than 100GB disk space". We know the second pref was "Leave at least 50GB disk space free" because the user admitted that this was the problem. It also fits nicely with the BOINC error message of needing to "free 17889.5 MB". Here are the maths:-
40Gb - 7.8GB = 32.2GB free space. 50 - 32.2 = 17.8GB which is the shortfall in free space with the unfortunate 50GB setting. As you can see, this has nothing to do with the maximum disk space setting.
Of course there are. These days the cheapest drives to buy are 200GB. However, I strongly dispute the first part of the quote. I've had machines running for months with disk sizes between 2GB and 200GB and a preference of 10GB. I've just checked the BOINC folder on a machine running EAH and LHC on which BoincView is also installed as a subfolder. The BOINC folder is 65MB. Part of this is BoincView at 34.4MB (logs 33.8MB - ouch!!) so actually BOINC is about 0.03GB. Hey, what's wrong with that stupid SOB!! Doesn't it realise I've given it permission to use 10GB?? Why is it only using 0.03GB :).
Over the last two days, I've looked at quite a few (20 - 100, I've really lost track) reports of disk usage problems, going back to 2004. Many are obviously the result of user error - just like the one you quoted. Many don't have enough data - don't you just hate it when you spend time trying to work it out and then ask for more information, only to be told, "Don't worry, I made a few random changes - don't remember what I did - seems to be sorted now, bye bye!!" :)
I've reached the conclusion that there is some "funny business" with that third preference setting but that the first two are NOT an issue. You can set the first one to whatever you want as long as it is large enough for future expansion and it doesn't matter about the actual partition size. You will, of course, be limited to that. You should set the second one basically to zero since no other software package seems to worry too much about running out of disk space :).
The third one is quite an enigma and the more I look, the more it seems that someone should really look closely at the code where it is used because there may be something funny going on. Unfortunately, that's not going to be me - programmer I ain't :).
Cheers,
Gary.
Bruce Allen and Walt Gribben
)
Bruce Allen and Walt Gribben have recent posts here that may shed some light on the problems described here and on similar threads. However, since we don't have a whole lot of data on the problem, I am not sure that their admissions clear up everything. I guess time will tell.
Hi Stick, I've been away
)
Hi Stick,
I've been away for a couple of days and I check back and it seems like all hell's broken loose. I see that you, Michael, Bill, Walt, Kathryn and others have been doing an amazing job with handling the influx of new users. I've seen the rapid rollout of about 5 new BOINC versions to handle the problems revealed by the upsurge in users particularly over at Seti, and I've seen the comments by Bruce and Walt that you refer to and also the message from Bruce about their awareness of the need to fix 4.79 issues. Hopefully that might go a small way towards answering the valid concerns you raise about having to continually direct people to the beta app.
I'm going to go ask for clarification from Walt about what bug he found and why it seemed like a good idea at the time to equate zero with 100MB rather than zero really being zero. If it really cant be zero then why not make it 0.5K
Anyway, thanks for the "heads up".
Cheers,
Gary.
RE: ...I've seen the rapid
)
Gary,
Add another version to the list - 5.2.13 looks to correct some more of these recently-surfaced issues.
I haven't quite been pulling my weight on the boards recently. My rig's gotten balky with the little warmup in weather here, and I've been loathe to give back any hard-won ground in the overclocking battle. Firefox and Einstein haven't been playing nice together, especially since I've gone to 1.5 release candidates (1, 2, now 3), but I think it's mostly a heat issue, with ambient temps back upto 18C.
(edit) - I'm back now, after a 5-minute breather. I wasn't finished with the post, but the rig has this funny habit of locking or shutting down and rebooting, and I've completely lost a couple of long, hard-to-compose would-be posts due to it's abberant behavior. The good news is that, miraculously, no Einstein work has been lost or corrupted.
microcraft
"The arc of history is long, but it bends toward justice" - MLK
Perhaps a review of how the
)
Perhaps a review of how the "available space" is calculated would help.
BOINC sends your preferences and disk usage numbers to the scheduler when it requests work. Thats what actually gets used, you can see them in the sched_request_einstein.phys.uwm.edu.xml file.
The scheduler does some calculations with the numbers and then uses the smallest result.
For "new" clients that report the space BOINC uses, it calculates:
"use no more than __GB" minus "disk space used by BOINC"
("use no more than __%" times "total disk space") minus "disk space used by BOINC"
"disk free space" minus "leave at least __GB free"
and keeps the smallest. (it turns your GB preference into bytes, and your % into a fraction, I just left those out to make it easier to read)
Also note that "leave at least __GB free" can't be smaller than .001 (1,000,000 bytes), if you set something smaller the scheduler uses .001 instead of your preference. That leaves some slack for administrative files (client_state, scheduler requests/replies, std*.txt, stuff like that)
Older BOINC clients don't send the space BOINC uses, so it only makes one calculation:
Thats it for determining how much it can use.
Once the "available space" is calculated, it makes sure theres enough space for the workunit.
While its building a list of workunits to send, the scheduler keeps track of the disk space needed by the workunit. It counts:
space needed for each file not already on the host
space needed to run the workunit (100MB or 100,000,000 bytes)
So the minimum space needed is 101 megabytes (101,000,000 bytes) plus the space needed for any files it has to sent. At least for Einstein@Home, other projects have other requirements.
If you check the messages from the scheduler, it'll tell you which preference generated the smallest "available disk space" value, and whether you even have that much available.
The message "Not enough disk space (only __ MB free for BOINC)." means you have some disk space BOINC can use, but its not enough to send a workunit.
The message "No disk space (YOU must free __ MB before BOINC gets space)." means you don't even have the minumum ("available disk space" is negative), thats how much you have to free up just to get to zero.
And remember, you need 100MB more that that, and maybe another 10MB if it has to download new work files. Seti@Home only needs 500,000 bytes "working space" while CPDN needs more. Depends on the workunit.
Some notes on where things come from...
"Disk space" only uses the drive BOINC runs from. That info is written to BOINC's message log when it starts up, along with the total and free disk space. So for hosts with multiple partitions and multiple drives, it ignores all that free space on the "next partition over".
Current clients (BOINC 5.2) send the total disk space, free disk space and the disk space used by BOINC and all projects when it contacts the scheduler. So they're using current numbers - it gets the disk space info from the operating system and runs thru all the files and folder in the BOINC folder adding up whats used. Actual date for the change was October 4, so clients built after that should be sending the proper data.
Some earlier versions didn't always send the correct disk space, it read it at startup and kept sending the same numbers each time it contacted the scheduler. BOINC 4.19 was the last one that tracked disk space correctly that I know of, but it only sent the total disk space and free disk space. So BOINC versions 4.25 thru 5.1 may not be sending the correct space info.
If in doubt, look in the sched_request_einsten.phys.uwm.edu.xml file to see what actually got sent, then compare it to whats actually on your disk. If they're very different, and its stopping you from getting work, then perhaps its a good reason to update your BOINC client.
The tags are:
- total disk space
- free disk space
- total disk space used by BOINC
The preferences used by the scheduler come from your host, BOINC sends them in the scheduler request. So what you see in your preferences on the web site might not be what the scheduler uses. Check BOINC's message log to see what actually got sent and where they came from. Might actually be preferences from another project.
If in doubt, check the file sched_request.einstein.phys.uwm.edu,xml, they're in the " section.
There are times the scheduler uses its own defaults for preferences. Thats when BOINC doesn't have any to send, and there aren't any set for the user on the web site. New users attaching with the "create new account" option in BOINC have that problem, its an oversight thats being corrected.
Which is where the earlier "no disk space" problems came from, the defaults used by the scheduler were too small for the Einstein workunits, unless you had a very large drive.
Its easy to tell if you don't have preferences set, the "Preferences last modified" line near the top won't have a date.
(will add more later, this is all I have so far)
RE: Perhaps a review of how
)
Walt,
I'm extremely grateful for the time you have spent to document all this. It's a marvellous resource and it will take me a while to digest it all. It might have to wait a bit until after the flood of new users begins to subside, but it's something I really want to understand better than I currently do so I'm really grateful to have the information.
I was very interested to see that versions after 4.19 and before 5.2.x might be sending bad info as that would probably explain a number of puzzling reports that I've seen in the past. Thanks again for what you have done, and particularly for all the marvellous detail :).
Cheers,
Gary.
Thanks to Gary, Walt, and
)
Thanks to Gary, Walt, and others for the in-depth answers.
Regarding disk space usage preferences and remaining disk space: When the problem happened to me, I suppose the disk could have been very close to 50% full, and not quite enough left for BOINC and E@H to do another WU.... I can't imagine why I haven't seen the problem resurface since then as I have installed quite a bit of software since then (Microsoft antispyware, Ad Aware, Encyclopedia Britannica, virus and firewall updates, etc.), and haven't done any purposeful disk clean-up.
The disk is very close to 50% full now (just 1GB to go), so I'm going to see if I get the disk space error again. If so, I will come back to this thread with a report of the appropriate .xml files and results of any changes I make.
Very interesting discussion, and I hope I can provide some data.
Thanks!
RE: The disk is very close
)
Thank you for doing that!! A report back would be much appreciated, even if only to clarify the actual values that you can see if the problem happens to strike again.
Cheers,
Gary.
Just a little update: I got
)
Just a little update: I got tired of waiting for the "problem" to show up, so I duplicated a little more than a GByte of MP3 files. Two WUs have now downloaded.
sched_request.einstein.phys.uwm.edu.xml does report correctly. So now more than 50% of the disk is used up. No disk space errors yet.
RE: Just a little update:
)
Whats using the over 50% of the disk? If its BOINC then theres a problem that has to be fixed. If its something else, thats not a BOINC issue - the preference says how much disk space BOINC (and the BOINC applications) can use.
You can test this yourself by setting the percentage to match what BOINC actually uses on the disk. But don't do this while you're processing workunits, or have workunits queued up. BOINC is likely to abort them with an "exceeded disk space" message. Clear out your work queue (all projects), then try testing. You can also test "leave at least __GB free" by setting the preference to your current free disk space, and check the "Use at most..." by setting the preference to what BOINC actually uses. And remember to set them back to reasonable values when done testing.
Walt