no disk space

Jord
Joined: 26 Jan 05
Posts: 2952
Credit: 5893653
RAC: 34

RE: RE: I am just now

Message 19631 in response to message 19630

Quote:
Quote:
I am just now beginning to realize how much I don't understand about BOINC.

I would now like to amend that to read: I am amazed at how much I don't understand about BOINC.


What I don't know about BOINC or what I don't understand correctly, or what I think I know about BOINC but still would like to know better... all of that I learn from the Wiki. The Link is in my sig, Stick. Might be time you bookmarked it and then earmarked its pages. ;)

Stick
Stick
Joined: 24 Feb 05
Posts: 790
Credit: 33137945
RAC: 1033

RE: RE: RE: I am just

Message 19632 in response to message 19631

Quote:
Quote:
Quote:
I am just now beginning to realize how much I don't understand about BOINC.

I would now like to amend that to read: I am amazed at how much I don't understand about BOINC.


What I don't know about BOINC or what I don't understand correctly, or what I think I know about BOINC but still would like to know better... all of that I learn from the Wiki. The Link is in my sig, Stick. Might be time you bookmarked it and then earmarked its pages. ;)

Point taken!

Gary Roberts
Gary Roberts
Moderator
Joined: 9 Feb 05
Posts: 5872
Credit: 117906834677
RAC: 34555187

I'm sorry it took so long to

I'm sorry it took so long to get back to everyone on this.

Yesterday was my Mother-in-Law's 95th birthday. I promised my wife that I would come home early to help her with all the preparations since we were throwing this big party with 4 generations of relatives attending, the youngest being my new granddaughter who is only six weeks old and a real cutie :). I had hoped to sneak away early and compose the explanation based on my own direct experience but I wasn't able to do that. I'm doing it now and will post again when it's finished.

One final point, I've had a very quick look through what has been written and Jord is spot on with his comments about separating clearly in your mind, the separate functions of BOINC Manager (just a GUI) and the BOINC daemon (boinc.exe) which does all the real work and implements the actions which are simply monitored and reported by the GUI. Sometimes problems occur because people shut down the GUI which doesn't necessarily shut down the daemon. I don't believe it's a problem in this particular thread but it's always something to keep in mind when you are making changes. Always confirm that the daemon is indeed inactive. Don't assume it is because you shut down the Manager.

Cheers,
Gary.

Gary Roberts
Gary Roberts
Moderator
Joined: 9 Feb 05
Posts: 5872
Credit: 117906834677
RAC: 34555187

To all people having problems

To all people having problems with disk space:

There are three key preference settings about this which you should check through the "View or edit general preferences" link on your account page on the project website. If you are attached to multiple projects, you can make changes on any of the project websites and over time those changes will appear on all websites through which you have an active project. You have to "Update" changes on the website (otherwise they wont "stick") and you have to "Update" your own BOINC client, otherwise the change will be delayed until the next server/client communication. Please note that you cannot ultimately have different general preferences for things like disk space on the different projects to which your computer is attached unless you start using the various "venue" options of home, work, school, but that is beyond the intended scope of this discussion. The final point to fully understand is that the most recent change on any of the attached project websites will be the one that dominates.

The three key disk preference settings are:-

  • * Use no more than XX GB disk space.
    This is an absolute limit and NOT an invitation for BOINC to overuse. If BOINC needs to use 100MB then that's exactly what it will use, irrespective of whether or not you give it permission to use a whole lot more. It is entirely safe to set it to 10GB or even 100GB even if you only have a 2GB disk, as I have in one particular case. BOINC is always limited to the physical size of the partition on which it is installed anyway.

So why do I choose to set mine to 10GB? Simply because, at some time in the future, I may well choose to go beserk and start supporting every single project out there. I don't want to have to remember that I set this puny limit of say 0.5GB because that's all I had available at the time. I'm confident that BOINC is not going to abuse my disk space so why set some restrictively low limit. Can you think of any other major software package (take MS Office for example) that actually gives you a choice to limit its disk usage?

So, unless you have a very good reason to do otherwise, I suggest that you are less likely to ever shoot yourself in the foot by setting this value to something large enough for possible future expansion.

* Leave at least XX GB disk space free. This is also an absolute limit. I make the same point again. What other software package actually offers you the choice of having the software cease and desist if the disk space starts getting a little on the low side? Usually this is an OS function. The OS starts getting stroppy if you look like consuming all the available free space. So once again, since BOINC really isn't going to consume more than it really needs anyway, just set this to a fairly minimal value like 0.1GB or even less if you don't have much free space anyway.

* Use no more than XX% of total disk space. This one is the killer and is not well understood at all, including by me. We really need someone familiar with the code to come forward and really set out exactly what this does. I can't give you a precise answer but I can tell you the results of a few experiments I conducted about 6 months ago when I had a machine that was actually successfully completing work but getting "Client errors" because of disk space issues.

Firstly let me tell you what I *think* it means. Since BOINC is only ever interested in the partition on which it is installed, "total disk space" refers to the size of that partition, and may well be less that the size of the physical disk. So when you have say a 4GB partition and the default is 50% you may well think that this preference is saying that BOINC has permission to use up to 50% (ie 2GB) on this 4GB partition. I am reasonably convinced that this is NOT what will happen.

I believe that the preference is saying that BOINC plus anything else is only allowed to consume up to 2GB and that BOINC therefore cannot proceed unless there is still 2GB free.

Others have suggested that "total disk space" might refer to total free disk space at the time BOINC is installed. So, as an example, again consider our 4GB partition that has say 3GB free when we go to install BOINC. A preference of 50% could therefore mean that BOINC could be limited to 1.5GB. However, once again, I believe it is the reverse to this that is important. I believe the preference is effectively saying that BOINC cannot proceed unless there is still 1.5GB free.

Now I don't really know whether the preference setting (eg 50%) is referring to "total disk space" or "total free disk space", because with many of my systems I've been installing in a clean partition anyway and therefore the two are the same thing. However I am reasonably convinced that we should not be looking at the disk space that is used up but rather we should consider the free space left. If we set that preference to 50% then the balance (also 50%) must remain free or else BOINC will refuse to work. That means that anyone installing in a 20GB partition must leave 10GB free (for a preference setting of 50%) at all times. So, if other things start filling up that free space, BOINC, for no apparent reason might suddenly stop working. I think it is simply this that causes most people to start having problems. They have that preference set too low and are not aware of other things that are gradually consuming disk space. If we are prepared for BOINC to work with an absolute value of 0.1GB, we should set the requirement for the % of the total disk that can be used to a very high value - say 90 - 95%. That way we are only forcing BOINC to keep 5 - 10% free.

Now for the actual example that happened to me 6 months ago. I had a machine whose C:\\ partition where BOINC was installed had a total size of 3.2GB The free space was just over 1.3GB if I remember correctly. The three preferences that I had set were 10GB for max size, 0.2GB for Min Free and 60% of total disk. At the time, my belief was that 60% meant that BOINC could use 0.6 X 3.2 = 1.92GB. I soon started having occasional disk write problems so I changed the 60% to 70%. This cured the problems and I thought no more about it until a few weeks later when the problems started coming back. It wasn't until I noticed that other things had been using up the disk and whilst BOINC was still the same, the free space was now less than 1GB.

The machine didn't belong to me and was being used for other purposes so I wasn't able to really play around with it too much. However I felt it was too much of a coincidence that the problems started firstly when the free space dropped below 1.3 GB = 40% (pref @ 60%) and secondly when it dropped below 1.0 GB = 30% (pref @ 70%). I've since upped the pref to an even larger value and had no further problem.

Of course, all this might be pure coincidence and since I really wasn't able to investigate thoroughly, and didn't keep accurate notes, my memory might be a bit faulty anyway. It would be very illuminating to hear from someone who is familiar with the code. With the current problems, I don't think there is any disk corruption or faulty .xml files. I would be really interested to know if the problems disappear with pref settings of 10GB, 0.1GB and 90% respectively.

So, could all people actually having disk space related problems please set that third preference in particular to say 90 - 95% and see if the problems cease, thanks.

Cheers,
Gary.

Stick
Stick
Joined: 24 Feb 05
Posts: 790
Credit: 33137945
RAC: 1033

Gary, I am having trouble

Gary,

I am having trouble seeing how making these disc preference setting changes address the problems being reported. That is, I don't understand the connection these recommendations may have with the reports coming in which indicate the problem can be fixed by simply "forcing" existing general preferences to be reloaded.

Are you saying that BOINC can trigger one of these "no disc space" messages in the midst of calculations (maybe while temporarily needing more "scratch" disc space)? In other words, a temporary space crisis might occur, trigger the error message and then pass by the time the prefs are reloaded. Therefore, BOINC is able to work again ??? If this is the case, I would guess the temporary problem would tend to crop up again, eventually. (In fact, we may have a such a case already - see Guido on "Getting Started" - although it may be something else.) If I am off base with this speculaltion, can you elaborate a little on how the "refreshing of existing prefs" fits your scenario?

Bill Michael has a post on "Disk Space" (here), which essentially echoes your recommendation and it mentions reported similar problems on other projects. He also acknowledges the possibility of "bug". He didn't say which "other" projects. Maybe he will read this and let us know.

Anyway, I will greatly appreciate hearing your explanation.

Stick

Gary Roberts
Gary Roberts
Moderator
Joined: 9 Feb 05
Posts: 5872
Credit: 117906834677
RAC: 34555187

RE: I am having trouble

Message 19636 in response to message 19635

Quote:
I am having trouble seeing how making these disc preference setting changes address the problems being reported. That is, I don't understand the connection these recommendations may have with the reports coming in which indicate the problem can be fixed by simply "forcing" existing general preferences to be reloaded.

You are having trouble seeing the connection because there is none :). I believe that it was pure chance that the reloading of unchanged preferences seemed to cure (temporarily) the problem. Imagine the following scenario:-

Let's say the preference is to allow BOINC to use 60% of the total disk space and that space is 1.0GB. BOINC cannot work unless 0.4GB is free. Let's say that the current free space is 0.41GB. If no other process is creating temp files then everything continues on happily.

At one point, without user intervention, the user's computer decides to automatically download and install a "critical" update from Microsoft, or perhaps one of a myriad other addon components decides it's time for a new version. I'm just using these things as examples of what could happen. At the height of this updating or upgrading, 20MB of temp files are created and the free space is now only 0.39GB. Is the BOINC client going to have a problem? Well no, probably not as long as the temp files have been deleted and the free space is back to 0.41GB before the server needs to check things out.

Please bear in mind that it's the server that is making the decision that the client machine doesn't have enough free space, and not the client. The client tells the server what the situation is when the client is asking the server for more work and the server wants to do a check. I believe this is why you can get these apparently random, "Now it's OK" and a little later "Now it's not OK" type of events. I've actually seen this happen and been extremely puzzled about how it can flip from "OK" one minute to "Not OK" some time later when BOINC hasn't done anything. The above is my rationalization of the behaviour.

Quote:
Are you saying that BOINC can trigger one of these "no disc space" messages in the midst of calculations (maybe while temporarily needing more "scratch" disc space)?

Yes, anything, BOINC, science app or totally unrelated app that is using temp space at just the "wrong" moment could trigger a problem that then just as suddenly seems to go away. It actually can't quite be as simple as I've rationalized it because one of the problems was the science app failing to write out the final answers because of disk shortage right at the end. The local client must be doing a check at that time too.

Quote:
In other words, the temporary space crisis may have passed by the time the prefs are reloaded, and, therefore, BOINC is able to work again. If this is the case, I would guess the temporary problem would tend to crop up again, eventually.

Yes, yes, yes - it's happened to me and it really throws you into a loop - it's sooooo frustrating. Now I don't really know that I'm right but once I made the changes I suggested, it has not happened again to me in nearly six months now.

Quote:
(In fact, we may have a such a case already - see Guido on "Getting Started" - although it may be something else.) If I am off base with this speculaltion, can you elaborate a little on how the "refreshing of existing prefs" fits your scenario?

I haven't been back to look at Guido's case yet so I will shortly.

Quote:
Bill Michael has a post on "Disk Space" (here), which essentially echoes your recommendation and it also mentions reported problems on other projects. He also acknowledges the possibility of "bug". He didn't say which "other" projects. Maybe he will read this and let us know.

I don't think I've caught up with what Bill has to say yet so I'll seek it out too.

Quote:

Anyway, I will greatly appreciate hearing your explanation.

Stick

Thanks very much for your interest. I'm not sure I have the full story yet but it's one of those things that's been around for quite a while and I've not seen anyone come up with a really complete explanation. I actually did a search on the Seti boards for "disk preferences" and came up with a staggering number of hits where people were reporting this same error. The complaint was basically that "BOINC says I don't have enough disk space but I've got an XXGB disk and there is more than YYGB free - WTF!!!!" People just don't realise that 50% of an XXGB disk is a *lot* of space that BOINC must see as being free. That's not even counting the cases where BOINC is in quite a small partition with little free space on an otherwise reasonably empty, large physical disk.

Cheers,
Gary.

Stick
Stick
Joined: 24 Feb 05
Posts: 790
Credit: 33137945
RAC: 1033

RE: You are having trouble

Message 19637 in response to message 19636

Quote:
You are having trouble seeing the connection because there is none :). I believe that it was pure chance that the reloading of unchanged preferences seemed to cure (temporarily) the problem.

Gary,

"Connection" was a poor choice of words. I was really looking for an explanation that would allow for your "disc settings problem" hypothesis to coexist with the "reload existing settings" experiences that have been reported. I understand now. Thank you!

Unfortunately, we seem to be back at square one. That is, two theories ("settings problem" and "bug") and no hard evidence yet to prove (or disprove) either. I guess we should start looking for "repeat offender" reports. If we start getting some from the "refresh existing prefs" crowd, that would tend to bolster your "settings problem" theory. But, if we get any from the "changed to new recommended settings" group, that would bolster the "bug" idea. Do you agree with my logic?

In the meantime (and I think I might have asked this earlier) is there any specific information we should be asking for if and when additional new reports come in?

Stick

Stick
Stick
Joined: 24 Feb 05
Posts: 790
Credit: 33137945
RAC: 1033

RE: In the meantime (and I

Message 19638 in response to message 19637

Quote:
In the meantime (and I think I might have asked this earlier) is there any specific information we should be asking for if and when additional new reports come in?

Nevermind, I see you have already provided an excellent example of what to ask for in Message 21578 on Message boards : Problems and Bug Reports : Disk Space.

Tern
Tern
Joined: 27 Jul 05
Posts: 309
Credit: 99440614
RAC: 0

RE: Bill Michael has a post

Message 19639 in response to message 19635

Quote:
Bill Michael has a post on "Disk Space" (here), which essentially echoes your recommendation and it mentions reported similar problems on other projects. He also acknowledges the possibility of "bug". He didn't say which "other" projects. Maybe he will read this and let us know.

Predictor, SETI, and I think the BOINC boards. Not on SZTAKI, and I don't _think_ I've seen it on Rosetta. I have trouble keeping up with what question shows up where. You guys are doing all the work here and I'm just parroting what you come up with as it pops up elsewhere! :-) If I've echoed you here, either I didn't remember where I was or it was a newly started thread at some hour that I figured nobody else would be around.

I haven't tried to reproduce the problem myself, that may be the next step. I can afford to lose a couple of the short Rosetta WUs my Windows box is crunching...

Jord
Joined: 26 Jan 05
Posts: 2952
Credit: 5893653
RAC: 34

RE: The three key disk

Message 19640 in response to message 19634

Quote:
The three key disk preference settings are:-
  • * Use no more than XX GB disk space.
    This is an absolute limit and NOT an invitation for BOINC to overuse. If BOINC needs to use 100MB then that's exactly what it will use, irrespective of whether or not you give it permission to use a whole lot more. It is entirely safe to set it to 10GB or even 100GB even if you only have a 2GB disk, as I have in one particular case. BOINC is always limited to the physical size of the partition on which it is installed anyway.


Try this again, Gary. Here's a nice example. :)

If not changed and left to its own devices, BOINC will always try to take 100GB. Yes, there are drives bigger than that already. :)

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.