Some projects don't create checkpoint

Gary, Thank you for your very clear and detailed explanation of the log file outputs. And also your rough estimation of computation time between checkpoints of 20..30 minutes for "djejoine's" machine.

The problem may not be the checkpoint frequency of probably 20..30 minutes (Gary's estimate), but e.g. too many (12?) parallel tasks using way too much memory (BOINC configuration: "Memory: when computer is in use, use at most: xx % of memory"; and "use at most yy % of the processors"). Maybe virtual memory is used here, way larger than the physical memory... eventually continuously swapping to HDD/SSD? One could hear rattling noise of HDDs in the good old times then, but not with SSDs anymore. 

Your computer offers eight physical cores (16 virtual ones through hyper threading). It has 16 GiB RAM which allows to run maybe four... maximum five O3MD1 tasks in parallel (up to 2 GiB each) when running Windows 11. It's never possible to run 12 tasks of O3MD1 on this machine in parallel, maybe a mix of very few O3MD1 and further FGRP5 (700 MiB each), and BRP4X64 (200 MiB each).

In some GPU configuration thread (and in BOINC's client documentation) it is explained how one can specify for certain science apps that only a maximum number of parallel tasks of this type (e.g. O3MD1 CPU) should run.

e.g. creating a file app_config.xml into the einstein@home directory under .../ProgramData/BOINC/projects/einstein.phys.uwm.edu/  containing at least (e.g. limiting to three O3MD1 tasks in parallel):

<app_config>
  <app>
    <name>einstein_O3MD1</name>
    <max_concurrent>3</max_concurrent>
  </app>
</app_config>

That's interesting. In all my finished tasks which I checked (all doing 32 checkpoints) the final stage took less than 10 minutes on an old machine (Core i7-4770) which runs throttled (to about 60% of maximum speed via tool 'TThrottle').

The example task you mentioned is from an Apple host with Apple's own "M1" CPU. The science app logs:

einstein_O3MD1_1.03_x86_64-apple-darwin__GW-SSE2

which is an Intel x64 app for Darwin. That means there had to be some Apple magic layer emulating x86 code to run on Apple's M1 CPU (which is some type of ARM64, I assume). Maybe that's the reason for ~50 minutes in final stage.

No, you had 14 sub-loops towards the first checkpoint.  There is no 'c' character at the end of the line of dots so there were more sub-loops needed before a checkpoint could be created.  Unfortunately, for this example, all the calculations represented by the 14 dots have been lost.

Unfortunately, this is entirely as expected.

In an earlier response to other suggestions, you had said that you would try the 'hibernation' mechanism as a low power method for saving state and not losing all that progress.  If you really need to continue shutting down your machine at regular intervals, hibernation is probably the best solution.  Otherwise, you will continue to lose lots of progress, exactly as this example shows.

In case you're not aware of the 'task properties' feature of BOINC Manager, here is what you can do.  Pick a running task on the tasks list to select it.  Click the properties button in the side bar to see a full list of properties for that task.  I'm not running any CPU tasks, so I can't show you a relevant example.  I've chosen a GPU task to show an abbreviated list (some lines omitted).  I'm using BOINC version 7.16.11.  The properties display could be a bit different with different BOINC versions.

       Application                Gamma-ray pulsar binary search #1 on GPUs 1.18 (FGRPopencl1K-ati)<br />
       Resources                  0.3 CPUs + 0.5 AMD/ATI GPUs<br />
       CPU time                   00:00:16<br />
       CPU time since checkpoint  00:00:02<br />
       Elapsed time               00:04:54<br />
       Estimated time remaining   00:11:58<br />
       Fraction done              26.834%<br />

My checkpoints occur at ~60s intervals - the BOINC default. - and the CPU time component is very small (just 16s in nearly 5 mins) for these GPU tasks.  The next checkpoint for the above would happen just after 5 mins elapsed when the CPU time since checkpoint would revert to zero.

I'm not at all suggesting that checking properties is a viable method for timing a shut down.  It's absolutely not.  I'm pointing it out just as a technique you could use while experimenting, simply to verify if there is a recent checkpoint or not.

The logfile output is misleading if you are new to O3MD1 CPU tasks. Whenever the O3MD1 app is started it begins to write into the logfile. If it's started the first time for a task the logfile is empty. The first string written into the logfile after app startup always is:

putenv 'LAL_DEBUG_LEVEL=3'

If the science app runs for a while it logs a lot of preparation steps (reading input data, maybe already saved checkpoints...) until entering the analysis subroutine consisting of analysing a number of skypoints which number differs between (32 and maybe 64... the highest I have seen so far was 58). That's also the number of checkpoints for the whole task. After each completely processed skypoint, a checkpoint is written. Processing a single sykpoint (of total 32...64) runs a number of sub-loops. Each finished sub-loop is logged with a single dot ("."). The number of sub-loops within a skypoint differs depending on the number of skypoints in the task and task parameters. For my last finished tasks I observed:

a O3MD1 task containing 29 skypoints:

2023-03-02 09:24:23.9786 (15596) [normal]: Cpt:0,  total:29,  sky:1/1,  f1dot:1/29

.......................c

--> 23 sub-loops until checkpoint "c"

another task containing 32 skypoints:

2023-03-02 05:59:56.2958 (13348) [normal]: Cpt:0,  total:32,  sky:1/1,  f1dot:1/32

.......................c

--> also 23 sub-loops until checkpoint "c"

I also observed a task containing 35 skypoints doing 21 sub-loops per skypoint. So it differs from task to task.

Whenever a O3MD1 task is terminated which is either by exiting BOINC manually, shutting down the computer, OR maybe process is killed by the OS* because of missing memory. Then the science app terminates somewhere within running a sub-loop. That means all computation since the last checkpoint is lost. If no checkpoint had been written so far it will start again at 0% progress. This can also happen repeatedly if app is always terminated before checkpointing.

(* MS Windows' process scheduler kills large processes early when memory becomes very scarce)

When a previously terminated O3MD1 task is started again by BOINC, it will at first write the string "putenv 'LAL_DEBUG_LEVEL=3'" at the end of the existing logfile:

...putenv 'LAL_DEBUG_LEVEL=3'

That's in the last line after the last dot(s) of a previous task run. The three dots here, in front of "putenv..." now represent the lost computation effort from terminating the previous run of the app. Those (hopefully few) dots/sub-loops had to be computed again.

A successfully read checkpoint (13 of 32) after restarting a task looks like this:

2023-02-27 16:23:40.2548 (7444) [debug]: Successfully read checkpoint:13
2023-02-27 16:23:40.2848 (7444) [normal]: Cpt:13,  total:32,  sky:1/1,  f1dot:14/32
 
0.2023-02-27 16:23:40.2848 (7444) [normal]: CG:12877956 FG:1072940  f1dotmin_fg:-4.217282365576e-008 df1dot_fg:8.561347619048e-014 f2dotmin_fg:-3.495554761905e-019 df2dot_fg:3.495554761905e-020 f3dotmin_fg:0 df3dot_fg:1
.......

I'll give an example run and some time measurements later.

Your BRP7 (MeerKAT) tasks run on external ATI GPU card which has its own 16 GiB GPU memory. So these do not stress system RAM and they only consume CPU cycles part of the time (0.5 CPU can be seen as set by rule of thumb).

You should also try out different number of CPU tasks running in parallel (independent of discussed memory constraints). Your CPU has 8 physical cores, presenting 16 virtual ones to the OS. It depends on the current mix of tasks (O3MD1, FGRP5, BRP4, other BOINC projects) running in parallel if this hyper threading feature (2 virtual instead of 1 physical core) improves analysis throughput (earned credits) or if it adds overhead reducing throughput. You can measure task runtimes (yes they differ also between tasks...anyway) for a different number of tasks running in parallel. Sometimes its better to limit number of tasks in parallel even down to 50% (no hyper threading) to increase task troughput per day.

memory constraints:

I configured max 3 concurrent O3MD1 tasks (via app_config.xml) because I encountered memory allocation problems without limits for O3MD1. It solved my problems. Four or five O3MD1 tasks can be started manually in the BOINC manager, carefully, waiting some time between, but not automaticly by BOINC's scheduler. Even if there is just enough memory MS Windows sometimes rejects four or five of these O3MD1 processes trying to allocate 8..10 GiB at the same time. This often ends with at least one task erroring out with a mem alloc failure (wasting ALL computation done so far, deleting checkpoint and all files in task's slot directory immediately).

It's good to have such problem discussion here. It's a science project. Problems can only be discovered (or excluded) this way.

I think Einstein isn't very different than other projects. Only memory requirements of O3MD1 CPU tasks are challenging. So it's always a good idea to have a look at the computers process list and memory usage when running CPU task on large proportion of available (virtual) CPU cores. I don't know of other projects, but some FGRP5 and all O3MD1 tasks checkpoint rarely. Developers seem to have good reasons for this. We client users simply have to adapt (or disable such apps in the preferences).