What could be reason of such failure? (BRP Arecibo large)
5 Jul 2022 3:56:17 UTC
Topic 227778
(moderation:
TASK 1312278043
Name:p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1520_0
Workunit ID:651701089
Created:28 Jun 2022 8:39:14 UTC
Sent:28 Jun 2022 11:21:32 UTC
Report deadline:5 Jul 2022 11:21:32 UTC
Received:4 Jul 2022 20:21:10 UTC
Server state:Over
Outcome:Computation error
Client state:Compute error
Exit status:2 (0x00000002) Unknown error code
Computer:12835631
Run time (sec):26,878.45
CPU time (sec):26,401.47
Peak working set size (MB):218.8
Peak swap size (MB):214.95
Peak disk usage (MB):0.07
Validation state:Invalid
Granted credit:0
Application:Binary Radio Pulsar Search (Arecibo, large) v1.33 (BRP4X64)
windows_x86_64
Stderr output
<core_client_version>7.16.5</core_client_version> <![CDATA[ <message> The system cannot find the file specified. (0x2) - exit code 2 (0x2)</message> <stderr_txt> Activated exception handling... [06:19:56][10900][INFO ] Output file: '../../projects/einstein.phys.uwm.edu/p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1520_0_0' already exists - skipping pass [06:19:56][10900][INFO ] Output file: '../../projects/einstein.phys.uwm.edu/p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1520_0_1' already exists - skipping pass [06:19:56][10900][INFO ] Starting data processing... [06:19:56][10900][INFO ] Checkpoint file unavailable: p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1520.cpt (No such file or directory). ------> Starting from scratch... [06:19:56][10900][INFO ] Header contents: ------> Original WAPP file: ./p2030.20180911.G55.22+01.39.S.b0s0g0.00000_DM152.20 ------> Sample time in microseconds: 65.4762 ------> Observation time in seconds: 274.62705 ------> Time stamp (MJD): 58373.061282623414 ------> Number of samples/record: 0 ------> Center freq in MHz: 1214.289551 ------> Channel band in MHz: 0.336182022 ------> Number of channels/record: 960 ------> Nifs: 1 ------> RA (J2000): 192829.177999 ------> DEC (J2000): 202019.431601 ------> Galactic l: 0 ------> Galactic b: 0 ------> Name: G55.22+01.39.S ------> Lagformat: 0 ------> Sum: 1 ------> Level: 3 ------> AZ at start: 0 ------> ZA at start: 0 ------> AST at start: 0 ------> LST at start: 0 ------> Project ID: -- ------> Observers: -- ------> File size (bytes): 0 ------> Data size (bytes): 0 ------> Number of samples: 4194304 ------> Trial dispersion measure: 152.2 cm^-3 pc ------> Scale factor: 0.00139448 [06:20:01][10900][INFO ] Seed for random number generator is 1163925760. [06:20:07][10900][INFO ] Derived global search parameters: ------> f_A probability = 0.08 ------> single bin prob(P_noise > P_thr) = 1.32531e-008 ------> thr1 = 18.139 ------> thr2 = 21.241 ------> thr4 = 26.2686 ------> thr8 = 34.6478 ------> thr16 = 48.9581 [06:23:16][10900][INFO ] Checkpoint committed! [06:26:33][10900][INFO ] Checkpoint committed! [06:29:52][10900][INFO ] Checkpoint committed! [06:33:11][10900][INFO ] Checkpoint committed! [06:36:30][10900][INFO ] Checkpoint committed! [06:39:46][10900][INFO ] Checkpoint committed! [06:43:04][10900][INFO ] Checkpoint committed! [06:46:21][10900][INFO ] Checkpoint committed! [06:49:38][10900][INFO ] Checkpoint committed! [06:52:57][10900][INFO ] Checkpoint committed! [06:56:15][10900][INFO ] Checkpoint committed! [06:59:32][10900][INFO ] Checkpoint committed! [07:02:50][10900][INFO ] Checkpoint committed! [07:06:07][10900][INFO ] Checkpoint committed! [07:09:24][10900][INFO ] Checkpoint committed! [07:12:41][10900][INFO ] Checkpoint committed! [07:15:59][10900][INFO ] Checkpoint committed! [07:19:17][10900][INFO ] Checkpoint committed! [07:22:34][10900][INFO ] Checkpoint committed! [07:25:53][10900][INFO ] Checkpoint committed! [07:29:10][10900][INFO ] Checkpoint committed! [07:32:27][10900][INFO ] Checkpoint committed! [07:35:45][10900][INFO ] Checkpoint committed! [07:39:03][10900][INFO ] Checkpoint committed! [07:42:21][10900][INFO ] Checkpoint committed! [07:45:38][10900][INFO ] Checkpoint committed! [07:48:54][10900][INFO ] Checkpoint committed! [07:52:12][10900][INFO ] Checkpoint committed! [07:55:28][10900][INFO ] Checkpoint committed! [07:58:45][10900][INFO ] Checkpoint committed! [08:02:02][10900][INFO ] Checkpoint committed! [08:05:19][10900][INFO ] Checkpoint committed! [08:08:36][10900][INFO ] Checkpoint committed! [08:11:55][10900][INFO ] Checkpoint committed! [08:15:11][10900][INFO ] Checkpoint committed! [08:18:28][10900][INFO ] Checkpoint committed! [08:21:46][10900][INFO ] Checkpoint committed! [08:25:04][10900][INFO ] Checkpoint committed! [08:28:21][10900][INFO ] Checkpoint committed! [08:31:39][10900][INFO ] Checkpoint committed! [08:34:57][10900][INFO ] Checkpoint committed! [08:38:14][10900][INFO ] Checkpoint committed! [08:41:31][10900][INFO ] Checkpoint committed! [08:44:50][10900][INFO ] Checkpoint committed! [08:48:08][10900][INFO ] Checkpoint committed! [08:51:26][10900][INFO ] Checkpoint committed! [08:54:45][10900][INFO ] Checkpoint committed! [08:58:01][10900][INFO ] Checkpoint committed! [09:01:18][10900][INFO ] Checkpoint committed! [09:04:36][10900][INFO ] Checkpoint committed! [09:07:54][10900][INFO ] Checkpoint committed! [09:11:11][10900][INFO ] Checkpoint committed! [09:14:29][10900][INFO ] Checkpoint committed! [09:17:46][10900][INFO ] Checkpoint committed! [09:21:04][10900][INFO ] Checkpoint committed! [09:24:22][10900][INFO ] Checkpoint committed! [09:27:40][10900][INFO ] Checkpoint committed! [09:30:56][10900][INFO ] Checkpoint committed! [09:34:15][10900][INFO ] Checkpoint committed! [09:37:33][10900][INFO ] Checkpoint committed! [09:40:51][10900][INFO ] Checkpoint committed! [09:44:10][10900][INFO ] Checkpoint committed! [09:47:27][10900][INFO ] Checkpoint committed! [09:50:46][10900][INFO ] Checkpoint committed! [09:54:04][10900][INFO ] Checkpoint committed! [09:57:21][10900][INFO ] Checkpoint committed! [10:00:38][10900][INFO ] Checkpoint committed! [10:03:55][10900][INFO ] Checkpoint committed! [10:07:13][10900][INFO ] Checkpoint committed! [10:10:30][10900][INFO ] Checkpoint committed! [10:13:47][10900][INFO ] Checkpoint committed! [10:17:04][10900][INFO ] Checkpoint committed! [10:20:23][10900][INFO ] Checkpoint committed! [10:23:42][10900][INFO ] Checkpoint committed! [10:26:58][10900][INFO ] Checkpoint committed! [10:30:16][10900][INFO ] Checkpoint committed! [10:33:34][10900][INFO ] Checkpoint committed! [10:36:52][10900][INFO ] Checkpoint committed! [10:40:09][10900][INFO ] Checkpoint committed! [10:40:31][10900][INFO ] Statistics: count dirty SumSpec pages 553 (not checkpointed), Page Size 1024, fundamental_idx_hi-window_2: 329052 [10:40:31][10900][INFO ] Data processing finished successfully! [10:40:31][10900][INFO ] Starting data processing... [10:40:31][10900][INFO ] Checkpoint file unavailable: p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1520.cpt (No such file or directory). ------> Starting from scratch... [10:40:31][10900][INFO ] Header contents: ------> Original WAPP file: ./p2030.20180911.G55.22+01.39.S.b0s0g0.00000_DM152.30 ------> Sample time in microseconds: 65.4762 ------> Observation time in seconds: 274.62705 ------> Time stamp (MJD): 58373.061282621384 ------> Number of samples/record: 0 ------> Center freq in MHz: 1214.289551 ------> Channel band in MHz: 0.336182022 ------> Number of channels/record: 960 ------> Nifs: 1 ------> RA (J2000): 192829.177999 ------> DEC (J2000): 202019.431601 ------> Galactic l: 0 ------> Galactic b: 0 ------> Name: G55.22+01.39.S ------> Lagformat: 0 ------> Sum: 1 ------> Level: 3 ------> AZ at start: 0 ------> ZA at start: 0 ------> AST at start: 0 ------> LST at start: 0 ------> Project ID: -- ------> Observers: -- ------> File size (bytes): 0 ------> Data size (bytes): 0 ------> Number of samples: 4194304 ------> Trial dispersion measure: 152.3 cm^-3 pc ------> Scale factor: 0.00139649 [10:40:36][10900][INFO ] Seed for random number generator is 1166427085. [10:40:38][10900][INFO ] Derived global search parameters: ------> f_A probability = 0.08 ------> single bin prob(P_noise > P_thr) = 1.32531e-008 ------> thr1 = 18.139 ------> thr2 = 21.241 ------> thr4 = 26.2686 ------> thr8 = 34.6478 ------> thr16 = 48.9581 [10:43:26][10900][INFO ] Checkpoint committed! [10:46:43][10900][INFO ] Checkpoint committed! [10:50:01][10900][INFO ] Checkpoint committed! [10:53:17][10900][INFO ] Checkpoint committed! [10:56:34][10900][INFO ] Checkpoint committed! [10:59:53][10900][INFO ] Checkpoint committed! [11:03:09][10900][INFO ] Checkpoint committed! [11:06:26][10900][INFO ] Checkpoint committed! [11:09:44][10900][INFO ] Checkpoint committed! [11:13:01][10900][INFO ] Checkpoint committed! [11:16:18][10900][INFO ] Checkpoint committed! [11:19:36][10900][INFO ] Checkpoint committed! [11:22:52][10900][INFO ] Checkpoint committed! [11:26:09][10900][INFO ] Checkpoint committed! [11:29:27][10900][INFO ] Checkpoint committed! [11:32:44][10900][INFO ] Checkpoint committed! [11:36:01][10900][INFO ] Checkpoint committed! [11:39:19][10900][INFO ] Checkpoint committed! [11:42:38][10900][INFO ] Checkpoint committed! [11:45:55][10900][INFO ] Checkpoint committed! [11:49:12][10900][INFO ] Checkpoint committed! [11:52:28][10900][INFO ] Checkpoint committed! [11:55:45][10900][INFO ] Checkpoint committed! [11:59:02][10900][INFO ] Checkpoint committed! [12:02:21][10900][INFO ] Checkpoint committed! [12:05:39][10900][INFO ] Checkpoint committed! [12:08:57][10900][INFO ] Checkpoint committed! [12:12:15][10900][INFO ] Checkpoint committed! [12:15:33][10900][INFO ] Checkpoint committed! [12:18:50][10900][INFO ] Checkpoint committed! [12:22:08][10900][INFO ] Checkpoint committed! [12:25:26][10900][INFO ] Checkpoint committed! [12:28:44][10900][INFO ] Checkpoint committed! [12:32:00][10900][INFO ] Checkpoint committed! [12:35:17][10900][INFO ] Checkpoint committed! [12:38:33][10900][INFO ] Checkpoint committed! [12:41:52][10900][INFO ] Checkpoint committed! [12:45:09][10900][INFO ] Checkpoint committed! [12:48:27][10900][INFO ] Checkpoint committed! [12:51:45][10900][INFO ] Checkpoint committed! [12:55:04][10900][INFO ] Checkpoint committed! [12:58:20][10900][INFO ] Checkpoint committed! [13:01:38][10900][INFO ] Checkpoint committed! [13:04:56][10900][INFO ] Checkpoint committed! [13:08:14][10900][INFO ] Checkpoint committed! [13:11:31][10900][INFO ] Checkpoint committed! [13:14:48][10900][INFO ] Checkpoint committed! [13:18:05][10900][INFO ] Checkpoint committed! [13:21:23][10900][INFO ] Checkpoint committed! [13:24:41][10900][INFO ] Checkpoint committed! [13:27:58][10900][INFO ] Checkpoint committed! [13:31:15][10900][INFO ] Checkpoint committed! [13:34:33][10900][INFO ] Checkpoint committed! [13:37:50][10900][INFO ] Checkpoint committed! [13:41:08][10900][INFO ] Checkpoint committed! [13:44:25][10900][INFO ] Checkpoint committed! [13:47:42][10900][INFO ] Checkpoint committed! Activated exception handling... [13:48:40][9716][INFO ] Output file: '../../projects/einstein.phys.uwm.edu/p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1520_0_0' already exists - skipping pass [13:48:40][9716][INFO ] Output file: '../../projects/einstein.phys.uwm.edu/p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1520_0_1' already exists - skipping pass [13:48:40][9716][INFO ] Output file: '../../projects/einstein.phys.uwm.edu/p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1520_0_2' already exists - skipping pass [13:48:40][9716][INFO ] Output file: '../../projects/einstein.phys.uwm.edu/p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1520_0_3' already exists - skipping pass [13:48:40][9716][INFO ] Starting data processing... [13:48:40][9716][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1523.bin4 at template no. 4811 [13:48:40][9716][ERROR] Input file on command line ../../projects/einstein.phys.uwm.edu/p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1524.bin4 doesn't agree with input file ../../projects/einstein.phys.uwm.edu/p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1523.bin4 from checkpoint header. [13:48:40][9716][ERROR] Demodulation failed (error: 2)! 13:48:40 (9716): called boinc_finish </stderr_txt> ]]>
And from another result
)
And from another result (completed OK but too late to validate)
And there are few such "starting from scratch" in the log.
Some checkpointing issue?....
And another example of
)
And another example of re-restarting from scratch task:
https://einsteinathome.org/task/1312310946
Activated exception handling...
[12:11:53][9020][INFO ] Starting data processing...
[12:11:53][9020][INFO ] Checkpoint file unavailable: p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1520.cpt (No such file or directory).
------> Starting from scratch...
[12:11:53][9020][INFO ] Header contents:
------> Original WAPP file: ./p2030.20180911.G55.22+01.39.S.b0s0g0.00000_DM152.00
------> Sample time in microseconds: 65.4762
------> Observation time in seconds: 274.62705
------> Time stamp (MJD): 58373.061282627481
------> Number of samples/record: 0
------> Center freq in MHz: 1214.289551
------> Channel band in MHz: 0.336182022
------> Number of channels/record: 960
------> Nifs: 1
------> RA (J2000): 192829.177912:12:04 (9020): BOINC client no longer exists - exiting
Activated exception handling...
[17:34:48][9424][INFO ] Starting data processing...
[17:34:48][9424][INFO ] Checkpoint file unavailable: p2030.20180911.G57.41-02.51.S.b0s0g0.00000_976.cpt (No such file or directory).
------> Starting from scratch...
[17:34:48][9424][INFO ] Header contents:
------> Original WAPP file: ./p2030.20180911.G57.41-02.51.S.b0s0g0.00000_DM97.60
------> Sample time in microseconds: 65.4762
------> Observation time in seconds: 274.62705
------> Time stamp (MJD): 58373.066275624216
------> Number of samples/record: 0
------> Center freq in MHz: 1214.289551
------> Channel band in MHz: 0.336182022
------> Number of channels/record: 960
------> Nifs: 1
------> RA (J2000): 194735.686699
------> DEC (J2000): 202019.696899
------> Galactic l: 0
------> Galactic b: 0
------> Name: G57.41-02.51.S
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 4194304
------> Trial dispersion measure: 97.6 cm^-3 pc
------> Scale factor: 0.000753267
[17:34:52][9424][INFO ] Seed for random number generator is 1173940838.
[17:34:55][9424][INFO ] Derived global search parameters:
------> f_A probability = 0.08
------> single bin prob(P_noise > P_thr) = 1.32531e-008
------> thr1 = 18.139
------> thr2 = 21.241
------> thr4 = 26.2686
------> thr8 = 34.6478
------> thr16 = 48.9581
[17:38:07][9424][INFO ] Checkpoint committed!
[17:41:31][9424][INFO ] Checkpoint committed!
[17:44:49][9424][INFO ] Checkpoint committed!
[17:48:07][9424][INFO ] Checkpoint committed!
[17:51:24][9424][INFO ] Checkpoint committed!
[17:54:41][9424][INFO ] Checkpoint committed!
[17:58:00][9424][INFO ] Checkpoint committed!
[18:01:17][9424][INFO ] Checkpoint committed!
[18:04:34][9424][INFO ] Checkpoint committed!
[18:07:51][9424][INFO ] Checkpoint committed!
[18:11:09][9424][INFO ] Checkpoint committed!
[18:14:26][9424][INFO ] Checkpoint committed!
[18:17:45][9424][INFO ] Checkpoint committed!
[18:21:03][9424][INFO ] Checkpoint committed!
[18:24:21][9424][INFO ] Checkpoint committed!
[18:27:39][9424][INFO ] Checkpoint committed!
[18:30:57][9424][INFO ] Checkpoint committed!
[18:34:15][9424][INFO ] Checkpoint committed!
[18:37:32][9424][INFO ] Checkpoint committed!
[18:40:50][9424][INFO ] Checkpoint committed!
[18:44:06][9424][INFO ] Checkpoint committed!
[18:47:25][9424][INFO ] Checkpoint committed!
[18:50:43][9424][INFO ] Checkpoint committed!
[18:54:00][9424][INFO ] Checkpoint committed!
[18:57:18][9424][INFO ] Checkpoint committed!
[19:00:35][9424][INFO ] Checkpoint committed!
[19:03:52][9424][INFO ] Checkpoint committed!
[19:07:10][9424][INFO ] Checkpoint committed!
[19:10:28][9424][INFO ] Checkpoint committed!
[19:13:45][9424][INFO ] Checkpoint committed!
[19:17:03][9424][INFO ] Checkpoint committed!
[19:20:20][9424][INFO ] Checkpoint committed!
[19:23:38][9424][INFO ] Checkpoint committed!
[19:26:56][9424][INFO ] Checkpoint committed!
[19:30:12][9424][INFO ] Checkpoint committed!
[19:33:30][9424][INFO ] Checkpoint committed!
[19:36:48][9424][INFO ] Checkpoint committed!
[19:40:05][9424][INFO ] Checkpoint committed!
[19:43:23][9424][INFO ] Checkpoint committed!
[19:46:42][9424][INFO ] Checkpoint committed!
[19:49:58][9424][INFO ] Checkpoint committed!
[19:53:16][9424][INFO ] Checkpoint committed!
[19:56:34][9424][INFO ] Checkpoint committed!
[19:59:51][9424][INFO ] Checkpoint committed!
[20:03:08][9424][INFO ] Checkpoint committed!
[20:06:25][9424][INFO ] Checkpoint committed!
[20:09:41][9424][INFO ] Checkpoint committed!
[20:12:59][9424][INFO ] Checkpoint committed!
[20:16:17][9424][INFO ] Checkpoint committed!
[20:19:34][9424][INFO ] Checkpoint committed!
[20:22:51][9424][INFO ] Checkpoint committed!
[20:26:09][9424][INFO ] Checkpoint committed!
[20:29:27][9424][INFO ] Checkpoint committed!
[20:32:44][9424][INFO ] Checkpoint committed!
[20:36:01][9424][INFO ] Checkpoint committed!
[20:39:19][9424][INFO ] Checkpoint committed!
[20:42:35][9424][INFO ] Checkpoint committed!
[20:45:53][9424][INFO ] Checkpoint committed!
[20:49:11][9424][INFO ] Checkpoint committed!
[20:52:28][9424][INFO ] Checkpoint committed!
[20:55:45][9424][INFO ] Checkpoint committed!
[20:59:03][9424][INFO ] Checkpoint committed!
[21:02:23][9424][INFO ] Checkpoint committed!
[21:05:40][9424][INFO ] Checkpoint committed!
[21:08:58][9424][INFO ] Checkpoint committed!
[21:12:15][9424][INFO ] Checkpoint committed!
[21:15:32][9424][INFO ] Checkpoint committed!
[21:18:50][9424][INFO ] Checkpoint committed!
[21:22:09][9424][INFO ] Checkpoint committed!
[21:25:27][9424][INFO ] Checkpoint committed!
[21:28:46][9424][INFO ] Checkpoint committed!
[21:32:04][9424][INFO ] Checkpoint committed!
[21:35:22][9424][INFO ] Checkpoint committed!
[21:38:39][9424][INFO ] Checkpoint committed!
[21:41:56][9424][INFO ] Checkpoint committed!
[21:45:14][9424][INFO ] Checkpoint committed!
[21:48:31][9424][INFO ] Checkpoint committed!
[21:51:45][9424][INFO ] Statistics: count dirty SumSpec pages 4371 (not checkpointed), Page Size 1024, fundamental_idx_hi-window_2: 329052
[21:51:45][9424][INFO ] Data processing finished successfully!
[21:51:45][9424][INFO ] Starting data processing...
[21:51:45][9424][INFO ] Checkpoint file unavailable: p2030.20180911.G57.41-02.51.S.b0s0g0.00000_976.cpt (No such file or directory).
------> Starting from scratch...
[21:51:45][9424][INFO ] Header contents:
------> Original WAPP file: ./p2030.20180911.G57.41-02.51.S.b0s0g0.00000_DM97.70
------> Sample time in microseconds: 65.4762
------> Observation time in seconds: 274.62705
------> Time stamp (MJD): 58373.066275622186
------> Number of samples/record: 0
------> Center freq in MHz: 1214.289551
------> Channel band in MHz: 0.336182022
------> Number of channels/record: 960
------> Nifs: 1
------> RA (J2000): 194735.686699
------> DEC (J2000): 202019.696899
------> Galactic l: 0
------> Galactic b: 0
------> Name: G57.41-02.51.S
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 4194304
------> Trial dispersion measure: 97.7 cm^-3 pc
------> Scale factor: 0.000752802
[21:51:49][9424][INFO ] Seed for random number generator is 1173950925.
[21:51:52][9424][INFO ] Derived global search parameters:
------> f_A probability = 0.08
------> single bin prob(P_noise > P_thr) = 1.32531e-008
------> thr1 = 18.139
------> thr2 = 21.241
------> thr4 = 26.2686
------> thr8 = 34.6478
------> thr16 = 48.9581
[21:51:54][9424][INFO ] Checkpoint committed!
[21:55:11][9424][INFO ] Checkpoint committed!
[21:58:29][9424][INFO ] Checkpoint committed!
[22:01:47][9424][INFO ] Checkpoint committed!
[22:05:05][9424][INFO ] Checkpoint committed!
[22:08:21][9424][INFO ] Checkpoint committed!
[22:11:38][9424][INFO ] Checkpoint committed!
[22:14:56][9424][INFO ] Checkpoint committed!
[22:18:12][9424][INFO ] Checkpoint committed!
[22:21:31][9424][INFO ] Checkpoint committed!
[22:24:48][9424][INFO ] Checkpoint committed!
[22:28:06][9424][INFO ] Checkpoint committed!
[22:31:23][9424][INFO ] Checkpoint committed!
[22:34:42][9424][INFO ] Checkpoint committed!
[22:37:59][9424][INFO ] Checkpoint committed!
[22:41:18][9424][INFO ] Checkpoint committed!
[22:44:34][9424][INFO ] Checkpoint committed!
[22:47:52][9424][INFO ] Checkpoint committed!
[22:51:11][9424][INFO ] Checkpoint committed!
[22:54:30][9424][INFO ] Checkpoint committed!
[22:57:49][9424][INFO ] Checkpoint committed!
[23:01:06][9424][INFO ] Checkpoint committed!
[23:04:24][9424][INFO ] Checkpoint committed!
[23:07:42][9424][INFO ] Checkpoint committed!
[23:10:59][9424][INFO ] Checkpoint committed!
[23:14:17][9424][INFO ] Checkpoint committed!
[23:17:34][9424][INFO ] Checkpoint committed!
[23:20:50][9424][INFO ] Checkpoint committed!
[23:24:09][9424][INFO ] Checkpoint committed!
[23:27:25][9424][INFO ] Checkpoint committed!
[23:30:43][9424][INFO ] Checkpoint committed!
[23:34:00][9424][INFO ] Checkpoint committed!
[23:37:18][9424][INFO ] Checkpoint committed!
[23:40:35][9424][INFO ] Checkpoint committed!
[23:43:53][9424][INFO ] Checkpoint committed!
[23:47:10][9424][INFO ] Checkpoint committed!
[23:50:28][9424][INFO ] Checkpoint committed!
[23:53:45][9424][INFO ] Checkpoint committed!
[23:57:03][9424][INFO ] Checkpoint committed!
[00:00:21][9424][INFO ] Checkpoint committed!
[00:03:38][9424][INFO ] Checkpoint committed!
[00:06:55][9424][INFO ] Checkpoint committed!
[00:10:12][9424][INFO ] Checkpoint committed!
[00:13:30][9424][INFO ] Checkpoint committed!
[00:16:48][9424][INFO ] Checkpoint committed!
[00:20:05][9424][INFO ] Checkpoint committed!
[00:23:23][9424][INFO ] Checkpoint committed!
[00:26:40][9424][INFO ] Checkpoint committed!
[00:29:57][9424][INFO ] Checkpoint committed!
[00:33:14][9424][INFO ] Checkpoint committed!
[00:36:32][9424][INFO ] Checkpoint committed!
[00:39:49][9424][INFO ] Checkpoint committed!
[00:43:06][9424][INFO ] Checkpoint committed!
[00:46:24][9424][INFO ] Checkpoint committed!
[00:49:42][9424][INFO ] Checkpoint committed!
[00:52:59][9424][INFO ] Checkpoint committed!
[00:56:18][9424][INFO ] Checkpoint committed!
[00:59:36][9424][INFO ] Checkpoint committed!
[01:02:53][9424][INFO ] Checkpoint committed!
[01:06:11][9424][INFO ] Checkpoint committed!
[01:09:29][9424][INFO ] Checkpoint committed!
[01:12:45][9424][INFO ] Checkpoint committed!
[01:16:04][9424][INFO ] Checkpoint committed!
[01:19:23][9424][INFO ] Checkpoint committed!
[01:22:42][9424][INFO ] Checkpoint committed!
[01:25:59][9424][INFO ] Checkpoint committed!
[01:29:17][9424][INFO ] Checkpoint committed!
[01:32:33][9424][INFO ] Checkpoint committed!
[01:35:51][9424][INFO ] Checkpoint committed!
[01:39:09][9424][INFO ] Checkpoint committed!
[01:42:25][9424][INFO ] Checkpoint committed!
[01:45:44][9424][INFO ] Checkpoint committed!
[01:49:02][9424][INFO ] Checkpoint committed!
[01:52:18][9424][INFO ] Checkpoint committed!
[01:55:35][9424][INFO ] Checkpoint committed!
[01:58:52][9424][INFO ] Checkpoint committed!
[02:00:23][9424][INFO ] Statistics: count dirty SumSpec pages 4466 (not checkpointed), Page Size 1024, fundamental_idx_hi-window_2: 329052
[02:00:23][9424][INFO ] Data processing finished successfully!
[02:00:23][9424][INFO ] Starting data processing...
[02:00:23][9424][INFO ] Checkpoint file unavailable: p2030.20180911.G57.41-02.51.S.b0s0g0.00000_976.cpt (No such file or directory).
------> Starting from scratch...
[02:00:23][9424][INFO ] Header contents:
------> Original WAPP file: ./p2030.20180911.G57.41-02.51.S.b0s0g0.00000_DM97.80
------> Sample time in microseconds: 65.4762
------> Observation time in seconds: 274.62705
------> Time stamp (MJD): 58373.066275620149
------> Number of samples/record: 0
------> Center freq in MHz: 1214.289551
------> Channel band in MHz: 0.336182022
------> Number of channels/record: 960
------> Nifs: 1
------> RA (J2000): 194735.686699
------> DEC (J2000): 202019.696899
------> Galactic l: 0
------> Galactic b: 0
------> Name: G57.41-02.51.S
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 4194304
------> Trial dispersion measure: 97.8 cm^-3 pc
------> Scale factor: 0.000752651
[02:00:27][9424][INFO ] Seed for random number generator is 1171233152.
[02:00:29][9424][INFO ] Derived global search parameters:
------> f_A probability = 0.08
------> single bin prob(P_noise > P_thr) = 1.32531e-008
------> thr1 = 18.139
------> thr2 = 21.241
------> thr4 = 26.2686
------> thr8 = 34.6478
------> thr16 = 48.9581
[02:02:10][9424][INFO ] Checkpoint committed!
[02:05:28][9424][INFO ] Checkpoint committed!
[02:08:46][9424][INFO ] Checkpoint committed!
[02:12:04][9424][INFO ] Checkpoint committed!
[02:15:22][9424][INFO ] Checkpoint committed!
[02:18:38][9424][INFO ] Checkpoint committed!
[02:21:55][9424][INFO ] Checkpoint committed!
[02:25:11][9424][INFO ] Checkpoint committed!
[02:28:29][9424][INFO ] Checkpoint committed!
[02:31:47][9424][INFO ] Checkpoint committed!
[02:35:03][9424][INFO ] Checkpoint committed!
[02:38:21][9424][INFO ] Checkpoint committed!
[02:41:39][9424][INFO ] Checkpoint committed!
[02:44:56][9424][INFO ] Checkpoint committed!
[02:48:14][9424][INFO ] Checkpoint committed!
[02:51:32][9424][INFO ] Checkpoint committed!
[02:54:49][9424][INFO ] Checkpoint committed!
[02:58:06][9424][INFO ] Checkpoint committed!
[03:01:23][9424][INFO ] Checkpoint committed!
[03:04:40][9424][INFO ] Checkpoint committed!
[03:07:57][9424][INFO ] Checkpoint committed!
[03:11:15][9424][INFO ] Checkpoint committed!
[03:14:33][9424][INFO ] Checkpoint committed!
[03:17:52][9424][INFO ] Checkpoint committed!
[03:21:10][9424][INFO ] Checkpoint committed!
[03:24:27][9424][INFO ] Checkpoint committed!
[03:27:44][9424][INFO ] Checkpoint committed!
[03:31:01][9424][INFO ] Checkpoint committed!
[03:34:20][9424][INFO ] Checkpoint committed!
[03:37:36][9424][INFO ] Checkpoint committed!
[03:40:53][9424][INFO ] Checkpoint committed!
[03:44:12][9424][INFO ] Checkpoint committed!
[03:47:31][9424][INFO ] Checkpoint committed!
[03:50:48][9424][INFO ] Checkpoint committed!
[03:54:06][9424][INFO ] Checkpoint committed!
[03:57:23][9424][INFO ] Checkpoint committed!
[04:00:39][9424][INFO ] Checkpoint committed!
[04:03:57][9424][INFO ] Checkpoint committed!
[04:07:16][9424][INFO ] Checkpoint committed!
[04:10:33][9424][INFO ] Checkpoint committed!
[04:13:51][9424][INFO ] Checkpoint committed!
[04:17:09][9424][INFO ] Checkpoint committed!
[04:20:28][9424][INFO ] Checkpoint committed!
[04:23:44][9424][INFO ] Checkpoint committed!
[04:27:02][9424][INFO ] Checkpoint committed!
[04:30:20][9424][INFO ] Checkpoint committed!
[04:33:36][9424][INFO ] Checkpoint committed!
[04:36:54][9424][INFO ] Checkpoint committed!
[04:40:11][9424][INFO ] Checkpoint committed!
[04:43:27][9424][INFO ] Checkpoint committed!
[04:46:44][9424][INFO ] Checkpoint committed!
[04:50:01][9424][INFO ] Checkpoint committed!
[04:53:17][9424][INFO ] Checkpoint committed!
[04:56:36][9424][INFO ] Checkpoint committed!
[04:59:54][9424][INFO ] Checkpoint committed!
[05:03:11][9424][INFO ] Checkpoint committed!
[05:06:28][9424][INFO ] Checkpoint committed!
[05:09:46][9424][INFO ] Checkpoint committed!
[05:13:04][9424][INFO ] Checkpoint committed!
[05:16:21][9424][INFO ] Checkpoint committed!
[05:19:39][9424][INFO ] Checkpoint committed!
[05:22:58][9424][INFO ] Checkpoint committed!
[05:26:16][9424][INFO ] Checkpoint committed!
[05:29:35][9424][INFO ] Checkpoint committed!
[05:32:51][9424][INFO ] Checkpoint committed!
[05:36:10][9424][INFO ] Checkpoint committed!
[05:39:28][9424][INFO ] Checkpoint committed!
[05:42:46][9424][INFO ] Checkpoint committed!
[05:46:05][9424][INFO ] Checkpoint committed!
[05:49:22][9424][INFO ] Checkpoint committed!
[05:52:40][9424][INFO ] Checkpoint committed!
[05:55:58][9424][INFO ] Checkpoint committed!
[05:59:15][9424][INFO ] Checkpoint committed!
[06:02:34][9424][INFO ] Checkpoint committed!
[06:05:52][9424][INFO ] Checkpoint committed!
[06:09:11][9424][INFO ] Checkpoint committed!
[06:12:28][9424][INFO ] Checkpoint committed!
[06:15:47][9424][INFO ] Checkpoint committed!
[06:19:05][9424][INFO ] Checkpoint committed!
[06:22:23][9424][INFO ] Checkpoint committed!
[06:25:42][9424][INFO ] Checkpoint committed!
[06:27:47][9424][INFO ] Statistics: count dirty SumSpec pages 4561 (not checkpointed), Page Size 1024, fundamental_idx_hi-window_2: 329052
[06:27:47][9424][INFO ] Data processing finished successfully!
[06:27:47][9424][INFO ] Starting data processing...
[06:27:48][9424][INFO ] Checkpoint file unavailable: p2030.20180911.G57.41-02.51.S.b0s0g0.00000_976.cpt (No such file or directory).
------> Starting from scratch...
[06:27:48][9424][INFO ] Header contents:
------> Original WAPP file: ./p2030.20180911.G57.41-02.51.S.b0s0g0.00000_DM97.90
------> Sample time in microseconds: 65.4762
------> Observation time in seconds: 274.62705
------> Time stamp (MJD): 58373.066275618119
------> Number of samples/record: 0
------> Center freq in MHz: 1214.289551
------> Channel band in MHz: 0.336182022
------> Number of channels/record: 960
------> Nifs: 1
------> RA (J2000): 194735.686699
------> DEC (J2000): 202019.696899
------> Galactic l: 0
------> Galactic b: 0
------> Name: G57.41-02.51.S
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 4194304
------> Trial dispersion measure: 97.9 cm^-3 pc
------> Scale factor: 0.000752748
[06:27:52][9424][INFO ] Seed for random number generator is 1171231403.
[06:27:54][9424][INFO ] Derived global search parameters:
------> f_A probability = 0.08
------> single bin prob(P_noise > P_thr) = 1.32531e-008
------> thr1 = 18.139
------> thr2 = 21.241
------> thr4 = 26.2686
------> thr8 = 34.6478
------> thr16 = 48.9581
[06:29:00][9424][INFO ] Checkpoint committed!
[06:32:17][9424][INFO ] Checkpoint committed!
[06:35:34][9424][INFO ] Checkpoint committed!
[06:38:52][9424][INFO ] Checkpoint committed!
[06:42:10][9424][INFO ] Checkpoint committed!
[06:45:27][9424][INFO ] Checkpoint committed!
[06:48:44][9424][INFO ] Checkpoint committed!
[06:52:01][9424][INFO ] Checkpoint committed!
[06:55:18][9424][INFO ] Checkpoint committed!
[06:58:36][9424][INFO ] Checkpoint committed!
[07:01:53][9424][INFO ] Checkpoint committed!
[07:05:12][9424][INFO ] Checkpoint committed!
[07:08:31][9424][INFO ] Checkpoint committed!
[07:11:49][9424][INFO ] Checkpoint committed!
[07:15:06][9424][INFO ] Checkpoint committed!
[07:18:24][9424][INFO ] Checkpoint committed!
[07:21:42][9424][INFO ] Checkpoint committed!
[07:25:00][9424][INFO ] Checkpoint committed!
[07:28:17][9424][INFO ] Checkpoint committed!
[07:31:35][9424][INFO ] Checkpoint committed!
[07:34:52][9424][INFO ] Checkpoint committed!
[07:38:08][9424][INFO ] Checkpoint committed!
[07:41:26][9424][INFO ] Checkpoint committed!
[07:44:44][9424][INFO ] Checkpoint committed!
[07:48:01][9424][INFO ] Checkpoint committed!
[07:51:19][9424][INFO ] Checkpoint committed!
[07:54:35][9424][INFO ] Checkpoint committed!
[07:57:53][9424][INFO ] Checkpoint committed!
[08:01:12][9424][INFO ] Checkpoint committed!
[08:04:28][9424][INFO ] Checkpoint committed!
[08:07:45][9424][INFO ] Checkpoint committed!
[08:11:02][9424][INFO ] Checkpoint committed!
[08:14:18][9424][INFO ] Checkpoint committed!
[08:17:37][9424][INFO ] Checkpoint committed!
[08:20:55][9424][INFO ] Checkpoint committed!
[08:24:13][9424][INFO ] Checkpoint committed!
[08:27:30][9424][INFO ] Checkpoint committed!
[08:30:48][9424][INFO ] Checkpoint committed!
[08:34:07][9424][INFO ] Checkpoint committed!
[08:37:24][9424][INFO ] Checkpoint committed!
[08:40:42][9424][INFO ] Checkpoint committed!
[08:43:59][9424][INFO ] Checkpoint committed!
[08:47:16][9424][INFO ] Checkpoint committed!
[08:50:34][9424][INFO ] Checkpoint committed!
[08:53:51][9424][INFO ] Checkpoint committed!
[08:57:07][9424][INFO ] Checkpoint committed!
[09:00:24][9424][INFO ] Checkpoint committed!
[09:03:42][9424][INFO ] Checkpoint committed!
[09:06:59][9424][INFO ] Checkpoint committed!
[09:10:15][9424][INFO ] Checkpoint committed!
[09:13:33][9424][INFO ] Checkpoint committed!
[09:16:49][9424][INFO ] Checkpoint committed!
[09:20:07][9424][INFO ] Checkpoint committed!
[09:23:26][9424][INFO ] Checkpoint committed!
[09:26:43][9424][INFO ] Checkpoint committed!
[09:30:02][9424][INFO ] Checkpoint committed!
[09:33:18][9424][INFO ] Checkpoint committed!
[09:36:36][9424][INFO ] Checkpoint committed!
[09:39:53][9424][INFO ] Checkpoint committed!
[09:43:10][9424][INFO ] Checkpoint committed!
[09:46:28][9424][INFO ] Checkpoint committed!
[09:49:46][9424][INFO ] Checkpoint committed!
[09:53:05][9424][INFO ] Checkpoint committed!
[09:56:22][9424][INFO ] Checkpoint committed!
[09:59:39][9424][INFO ] Checkpoint committed!
[10:02:57][9424][INFO ] Checkpoint committed!
[10:06:14][9424][INFO ] Checkpoint committed!
[10:09:30][9424][INFO ] Checkpoint committed!
[10:12:47][9424][INFO ] Checkpoint committed!
[10:16:05][9424][INFO ] Checkpoint committed!
[10:19:24][9424][INFO ] Checkpoint committed!
[10:22:42][9424][INFO ] Checkpoint committed!
[10:25:59][9424][INFO ] Checkpoint committed!
[10:29:17][9424][INFO ] Checkpoint committed!
[10:32:35][9424][INFO ] Checkpoint committed!
[10:35:54][9424][INFO ] Checkpoint committed!
[10:39:13][9424][INFO ] Checkpoint committed!
[10:42:31][9424][INFO ] Checkpoint committed!
[10:45:49][9424][INFO ] Checkpoint committed!
[10:48:17][9424][INFO ] Statistics: count dirty SumSpec pages 4499 (not checkpointed), Page Size 1024, fundamental_idx_hi-window_2: 329052
[10:48:17][9424][INFO ] Data processing finished successfully!
[10:48:17][9424][INFO ] Starting data processing...
[10:48:17][9424][INFO ] Checkpoint file unavailable: p2030.20180911.G57.41-02.51.S.b0s0g0.00000_976.cpt (No such file or directory).
------> Starting from scratch...
[10:48:17][9424][INFO ] Header contents:
------> Original WAPP file: ./p2030.20180911.G57.41-02.51.S.b0s0g0.00000_DM98.00
------> Sample time in microseconds: 65.4762
------> Observation time in seconds: 274.62705
------> Time stamp (MJD): 58373.066275616082
------> Number of samples/record: 0
------> Center freq in MHz: 1214.289551
------> Channel band in MHz: 0.336182022
------> Number of channels/record: 960
------> Nifs: 1
------> RA (J2000): 194735.686699
------> DEC (J2000): 202019.696899
------> Galactic l: 0
------> Galactic b: 0
------> Name: G57.41-02.51.S
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 4194304
------> Trial dispersion measure: 98 cm^-3 pc
------> Scale factor: 0.000753225
[10:48:21][9424][INFO ] Seed for random number generator is 1168503808.
[10:48:24][9424][INFO ] Derived global search parameters:
------> f_A probability = 0.08
------> single bin prob(P_noise > P_thr) = 1.32531e-008
------> thr1 = 18.139
------> thr2 = 21.241
------> thr4 = 26.2686
------> thr8 = 34.6478
------> thr16 = 48.9581
[10:49:08][9424][INFO ] Checkpoint committed!
[10:52:25][9424][INFO ] Checkpoint committed!
[10:55:43][9424][INFO ] Checkpoint committed!
[10:59:01][9424][INFO ] Checkpoint committed!
[11:02:17][9424][INFO ] Checkpoint committed!
[11:05:34][9424][INFO ] Checkpoint committed!
[11:08:53][9424][INFO ] Checkpoint committed!
[11:12:11][9424][INFO ] Checkpoint committed!
[11:15:28][9424][INFO ] Checkpoint committed!
[11:18:44][9424][INFO ] Checkpoint committed!
[11:22:01][9424][INFO ] Checkpoint committed!
[11:25:19][9424][INFO ] Checkpoint committed!
[11:28:37][9424][INFO ] Checkpoint committed!
[11:31:55][9424][INFO ] Checkpoint committed!
[11:35:11][9424][INFO ] Checkpoint committed!
[11:38:28][9424][INFO ] Checkpoint committed!
[11:41:47][9424][INFO ] Checkpoint committed!
[11:45:04][9424][INFO ] Checkpoint committed!
[11:48:21][9424][INFO ] Checkpoint committed!
[11:51:37][9424][INFO ] Checkpoint committed!
[11:54:54][9424][INFO ] Checkpoint committed!
[11:58:11][9424][INFO ] Checkpoint committed!
[12:01:28][9424][INFO ] Checkpoint committed!
[12:04:46][9424][INFO ] Checkpoint committed!
[12:08:04][9424][INFO ] Checkpoint committed!
[12:11:21][9424][INFO ] Checkpoint committed!
[12:14:37][9424][INFO ] Checkpoint committed!
[12:17:55][9424][INFO ] Checkpoint committed!
[12:21:13][9424][INFO ] Checkpoint committed!
[12:24:31][9424][INFO ] Checkpoint committed!
[12:27:48][9424][INFO ] Checkpoint committed!
[12:31:05][9424][INFO ] Checkpoint committed!
[12:34:22][9424][INFO ] Checkpoint committed!
[12:37:40][9424][INFO ] Checkpoint committed!
[12:40:56][9424][INFO ] Checkpoint committed!
[12:44:13][9424][INFO ] Checkpoint committed!
[12:47:31][9424][INFO ] Checkpoint committed!
[12:50:50][9424][INFO ] Checkpoint committed!
[12:54:07][9424][INFO ] Checkpoint committed!
[12:57:25][9424][INFO ] Checkpoint committed!
[13:00:43][9424][INFO ] Checkpoint committed!
[13:04:02][9424][INFO ] Checkpoint committed!
[13:07:20][9424][INFO ] Checkpoint committed!
[13:10:37][9424][INFO ] Checkpoint committed!
[13:13:54][9424][INFO ] Checkpoint committed!
[13:17:12][9424][INFO ] Checkpoint committed!
[13:20:29][9424][INFO ] Checkpoint committed!
[13:23:47][9424][INFO ] Checkpoint committed!
[13:27:04][9424][INFO ] Checkpoint committed!
[13:30:21][9424][INFO ] Checkpoint committed!
[13:33:39][9424][INFO ] Checkpoint committed!
[13:36:58][9424][INFO ] Checkpoint committed!
[13:40:15][9424][INFO ] Checkpoint committed!
[13:43:33][9424][INFO ] Checkpoint committed!
[13:46:52][9424][INFO ] Checkpoint committed!
Activated exception handling...
[13:48:38][2104][INFO ] Output file: '../../projects/einstein.phys.uwm.edu/p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1520_0_0' already exists - skipping pass
[13:48:38][2104][INFO ] Output file: '../../projects/einstein.phys.uwm.edu/p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1520_0_1' already exists - skipping pass
[13:48:38][2104][INFO ] Output file: '../../projects/einstein.phys.uwm.edu/p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1520_0_2' already exists - skipping pass
[13:48:38][2104][INFO ] Output file: '../../projects/einstein.phys.uwm.edu/p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1520_0_3' already exists - skipping pass
[13:48:38][2104][INFO ] Starting data processing...
[13:48:38][2104][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/p2030.20180911.G57.41-02.51.S.b0s0g0.00000_980.bin4 at template no. 4614
[13:48:38][2104][INFO ] Header contents:
------> Original WAPP file: ./p2030.20180911.G57.41-02.51.S.b0s0g0.00000_DM98.00
------> Sample time in microseconds: 65.4762
------> Observation time in seconds: 274.62705
------> Time stamp (MJD): 58373.066275616082
------> Number of samples/record: 0
------> Center freq in MHz: 1214.289551
------> Channel band in MHz: 0.336182022
------> Number of channels/record: 960
------> Nifs: 1
------> RA (J2000): 194735.686699
------> DEC (J2000): 202019.696899
------> Galactic l: 0
------> Galactic b: 0
------> Name: G57.41-02.51.S
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 4194304
------> Trial dispersion measure: 98 cm^-3 pc
------> Scale factor: 0.000753225
[13:48:44][2104][INFO ] Seed for random number generator is 1168503808.
[13:48:48][2104][INFO ] Derived global search parameters:
------> f_A probability = 0.08
------> single bin prob(P_noise > P_thr) = 1.32531e-008
------> thr1 = 18.139
------> thr2 = 21.241
------> thr4 = 26.2686
------> thr8 = 34.6478
------> thr16 = 48.9581
[13:52:01][2104][INFO ] Checkpoint committed!
[13:55:20][2104][INFO ] Checkpoint committed!
[13:58:37][2104][INFO ] Checkpoint committed!
[14:01:53][2104][INFO ] Checkpoint committed!
[14:05:10][2104][INFO ] Checkpoint committed!
[14:08:29][2104][INFO ] Checkpoint committed!
[14:11:47][2104][INFO ] Checkpoint committed!
[14:15:04][2104][INFO ] Checkpoint committed!
[14:18:22][2104][INFO ] Checkpoint committed!
[14:21:40][2104][INFO ] Checkpoint committed!
[14:24:58][2104][INFO ] Checkpoint committed!
[14:28:16][2104][INFO ] Checkpoint committed!
[14:31:33][2104][INFO ] Checkpoint committed!
[14:34:51][2104][INFO ] Checkpoint committed!
[14:38:09][2104][INFO ] Checkpoint committed!
[14:41:21][2104][INFO ] Checkpoint committed!
[14:44:38][2104][INFO ] Checkpoint committed!
[23:21:19][2104][INFO ] Checkpoint committed!
[23:24:50][2104][INFO ] Checkpoint committed!
[23:28:20][2104][INFO ] Checkpoint committed!
[23:31:41][2104][INFO ] Checkpoint committed!
[00:34:20][2104][INFO ] Checkpoint committed!
[00:37:39][2104][INFO ] Checkpoint committed!
[00:40:55][2104][INFO ] Checkpoint committed!
[00:44:14][2104][INFO ] Checkpoint committed!
[00:45:06][2104][INFO ] Statistics: count dirty SumSpec pages 715 (not checkpointed), Page Size 1024, fundamental_idx_hi-window_2: 329052
[00:45:06][2104][INFO ] Data processing finished successfully!
[00:45:06][2104][INFO ] Starting data processing...
[00:45:06][2104][INFO ] Checkpoint file unavailable: p2030.20180911.G57.41-02.51.S.b0s0g0.00000_976.cpt (No such file or directory).
------> Starting from scratch...
[00:45:06][2104][INFO ] Header contents:
------> Original WAPP file: ./p2030.20180911.G57.41-02.51.S.b0s0g0.00000_DM98.10
------> Sample time in microseconds: 65.4762
------> Observation time in seconds: 274.62705
------> Time stamp (MJD): 58373.066275614052
------> Number of samples/record: 0
------> Center freq in MHz: 1214.289551
------> Channel band in MHz: 0.336182022
------> Number of channels/record: 960
------> Nifs: 1
------> RA (J2000): 194735.686699
------> DEC (J2000): 202019.696899
------> Galactic l: 0
------> Galactic b: 0
------> Name: G57.41-02.51.S
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 4194304
------> Trial dispersion measure: 98.1 cm^-3 pc
------> Scale factor: 0.000754124
[00:45:10][2104][INFO ] Seed for random number generator is 1171206571.
[00:45:12][2104][INFO ] Derived global search parameters:
------> f_A probability = 0.08
------> single bin prob(P_noise > P_thr) = 1.32531e-008
------> thr1 = 18.139
------> thr2 = 21.241
------> thr4 = 26.2686
------> thr8 = 34.6478
------> thr16 = 48.9581
[00:47:29][2104][INFO ] Checkpoint committed!
[00:50:46][2104][INFO ] Checkpoint committed!
[00:54:01][2104][INFO ] Checkpoint committed!
[00:57:17][2104][INFO ] Checkpoint committed!
[01:00:33][2104][INFO ] Checkpoint committed!
[01:03:48][2104][INFO ] Checkpoint committed!
[01:07:03][2104][INFO ] Checkpoint committed!
[01:10:19][2104][INFO ] Checkpoint committed!
[01:13:35][2104][INFO ] Checkpoint committed!
[01:16:52][2104][INFO ] Checkpoint committed!
[01:20:08][2104][INFO ] Checkpoint committed!
[01:23:24][2104][INFO ] Checkpoint committed!
[01:26:40][2104][INFO ] Checkpoint committed!
[01:29:56][2104][INFO ] Checkpoint committed!
[01:33:11][2104][INFO ] Checkpoint committed!
[01:36:26][2104][INFO ] Checkpoint committed!
[01:39:42][2104][INFO ] Checkpoint committed!
[01:42:58][2104][INFO ] Checkpoint committed!
[01:46:14][2104][INFO ] Checkpoint committed!
[01:49:29][2104][INFO ] Checkpoint committed!
[01:52:46][2104][INFO ] Checkpoint committed!
[01:56:02][2104][INFO ] Checkpoint committed!
[01:59:17][2104][INFO ] Checkpoint committed!
[02:02:32][2104][INFO ] Checkpoint committed!
[02:05:46][2104][INFO ] Checkpoint committed!
[02:09:01][2104][INFO ] Checkpoint committed!
[02:12:16][2104][INFO ] Checkpoint committed!
[02:15:31][2104][INFO ] Checkpoint committed!
[02:18:46][2104][INFO ] Checkpoint committed!
[02:22:02][2104][INFO ] Checkpoint committed!
[02:25:17][2104][INFO ] Checkpoint committed!
[02:28:32][2104][INFO ] Checkpoint committed!
[02:31:49][2104][INFO ] Checkpoint committed!
[02:35:04][2104][INFO ] Checkpoint committed!
[02:38:21][2104][INFO ] Checkpoint committed!
[02:41:36][2104][INFO ] Checkpoint committed!
[02:44:52][2104][INFO ] Checkpoint committed!
[02:48:08][2104][INFO ] Checkpoint committed!
[02:51:23][2104][INFO ] Checkpoint committed!
[02:54:38][2104][INFO ] Checkpoint committed!
[02:57:54][2104][INFO ] Checkpoint committed!
[03:01:09][2104][INFO ] Checkpoint committed!
[03:04:25][2104][INFO ] Checkpoint committed!
[03:07:42][2104][INFO ] Checkpoint committed!
[03:10:57][2104][INFO ] Checkpoint committed!
[03:14:11][2104][INFO ] Checkpoint committed!
[03:17:27][2104][INFO ] Checkpoint committed!
[03:20:42][2104][INFO ] Checkpoint committed!
[03:23:57][2104][INFO ] Checkpoint committed!
[03:27:13][2104][INFO ] Checkpoint committed!
[03:30:29][2104][INFO ] Checkpoint committed!
[03:33:44][2104][INFO ] Checkpoint committed!
[03:36:59][2104][INFO ] Checkpoint committed!
[03:40:13][2104][INFO ] Checkpoint committed!
[03:43:29][2104][INFO ] Checkpoint committed!
[03:46:43][2104][INFO ] Checkpoint committed!
[03:49:58][2104][INFO ] Checkpoint committed!
[03:53:13][2104][INFO ] Checkpoint committed!
[03:56:28][2104][INFO ] Checkpoint committed!
[03:59:42][2104][INFO ] Checkpoint committed!
[04:02:57][2104][INFO ] Checkpoint committed!
[04:06:14][2104][INFO ] Checkpoint committed!
[04:09:29][2104][INFO ] Checkpoint committed!
[04:12:44][2104][INFO ] Checkpoint committed!
[04:15:59][2104][INFO ] Checkpoint committed!
[04:19:15][2104][INFO ] Checkpoint committed!
[04:22:31][2104][INFO ] Checkpoint committed!
[04:25:46][2104][INFO ] Checkpoint committed!
[04:29:01][2104][INFO ] Checkpoint committed!
[04:32:16][2104][INFO ] Checkpoint committed!
[04:35:32][2104][INFO ] Checkpoint committed!
[04:38:47][2104][INFO ] Checkpoint committed!
[04:42:01][2104][INFO ] Checkpoint committed!
[04:44:23][2104][INFO ] Statistics: count dirty SumSpec pages 4404 (not checkpointed), Page Size 1024, fundamental_idx_hi-window_2: 329052
[04:44:23][2104][INFO ] Data processing finished successfully!
[04:44:23][2104][INFO ] Starting data processing...
[04:44:23][2104][INFO ] Checkpoint file unavailable: p2030.20180911.G57.41-02.51.S.b0s0g0.00000_976.cpt (No such file or directory).
------> Starting from scratch...
[04:44:23][2104][INFO ] Header contents:
------> Original WAPP file: ./p2030.20180911.G57.41-02.51.S.b0s0g0.00000_DM98.20
------> Sample time in microseconds: 65.4762
------> Observation time in seconds: 274.62705
------> Time stamp (MJD): 58373.066275612015
------> Number of samples/record: 0
------> Center freq in MHz: 1214.289551
------> Channel band in MHz: 0.336182022
------> Number of channels/record: 960
------> Nifs: 1
------> RA (J2000): 194735.686699
------> DEC (J2000): 202019.696899
------> Galactic l: 0
------> Galactic b: 0
------> Name: G57.41-02.51.S
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 4194304
------> Trial dispersion measure: 98.2 cm^-3 pc
------> Scale factor: 0.000752679
[04:44:27][2104][INFO ] Seed for random number generator is 1160123085.
[04:44:29][2104][INFO ] Derived global search parameters:
------> f_A probability = 0.08
------> single bin prob(P_noise > P_thr) = 1.32531e-008
------> thr1 = 18.139
------> thr2 = 21.241
------> thr4 = 26.2686
------> thr8 = 34.6478
------> thr16 = 48.9581
[04:45:18][2104][INFO ] Checkpoint committed!
[04:48:33][2104][INFO ] Checkpoint committed!
[04:51:50][2104][INFO ] Checkpoint committed!
[04:55:06][2104][INFO ] Checkpoint committed!
[04:58:20][2104][INFO ] Checkpoint committed!
[05:01:35][2104][INFO ] Checkpoint committed!
[05:04:51][2104][INFO ] Checkpoint committed!
[05:08:05][2104][INFO ] Checkpoint committed!
[05:11:22][2104][INFO ] Checkpoint committed!
[05:14:38][2104][INFO ] Checkpoint committed!
[05:17:54][2104][INFO ] Checkpoint committed!
[05:21:11][2104][INFO ] Checkpoint committed!
[05:24:25][2104][INFO ] Checkpoint committed!
[05:27:42][2104][INFO ] Checkpoint committed!
[05:30:56][2104][INFO ] Checkpoint committed!
[05:34:12][2104][INFO ] Checkpoint committed!
[05:37:27][2104][INFO ] Checkpoint committed!
[05:40:43][2104][INFO ] Checkpoint committed!
[05:43:59][2104][INFO ] Checkpoint committed!
[05:47:14][2104][INFO ] Checkpoint committed!
[05:50:29][2104][INFO ] Checkpoint committed!
[05:53:45][2104][INFO ] Checkpoint committed!
[05:57:02][2104][INFO ] Checkpoint committed!
[06:00:19][2104][INFO ] Checkpoint committed!
[06:03:37][2104][INFO ] Checkpoint committed!
[06:06:54][2104][INFO ] Checkpoint committed!
[06:10:10][2104][INFO ] Checkpoint committed!
[06:13:29][2104][INFO ] Checkpoint committed!
[06:16:45][2104][INFO ] Checkpoint committed!
[06:20:02][2104][INFO ] Checkpoint committed!
[06:23:20][2104][INFO ] Checkpoint committed!
[06:26:37][2104][INFO ] Checkpoint committed!
[06:29:54][2104][INFO ] Checkpoint committed!
[06:33:12][2104][INFO ] Checkpoint committed!
[06:36:29][2104][INFO ] Checkpoint committed!
[06:39:47][2104][INFO ] Checkpoint committed!
[06:43:06][2104][INFO ] Checkpoint committed!
[06:46:22][2104][INFO ] Checkpoint committed!
[06:50:03][2104][INFO ] Checkpoint committed!
[06:53:21][2104][INFO ] Checkpoint committed!
[06:56:40][2104][INFO ] Checkpoint committed!
[07:00:00][2104][INFO ] Checkpoint committed!
[07:03:19][2104][INFO ] Checkpoint committed!
[07:06:37][2104][INFO ] Checkpoint committed!
[07:10:01][2104][INFO ] Checkpoint committed!
[07:13:18][2104][INFO ] Checkpoint committed!
Check with Richard Haselgrove
)
Check with Richard Haselgrove about your cpu. I believe there is a documented issue with that particular cpu with BOINC.
I would say that it hung up on the task.
Knew I remembered an issue with that cpu.
https://github.com/BOINC/boinc/issues/2470
https://boinc.n-helix.com/forum_thread.php?id=13526
Raistmer* wrote:... ------>
)
No.
Arecibo, large tasks are bundles of 8 smaller tasks. Each member of the bundle will "Start from scratch" at some point.
From the output in your first message, the error would appear to be:-
[13:48:40][9716][ERROR] Input file on command line
../../projects/einstein.phys.uwm.edu/p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1524.bin4
doesn't agree with input file
../../projects/einstein.phys.uwm.edu/p2030.20180911.G55.22+01.39.S.b0s0g0.00000_1523.bin4
from checkpoint header.
[13:48:40][9716][ERROR] Demodulation failed (error: 2)!
Notice the 1524 <--> 1523 mismatch in the data file name.
If you were stopping/restarting processing by any chance maybe something got out of sync?
My guess is that the smaller tasks that are part of the bundle of 8 have that final sequence number changing by one for each small task. It might be some sort of bug with how the stop/restart of processing occurs if it just happened to occur close to the changeover from one small task to the next. Maybe you just happened to hit the jackpot :-).
Cheers,
Gary.
The failure in his latest
)
The failure in his latest task he linked was Windows 10 blowing up.
C:\BOINCdata\projects\einstein.phys.uwm.edu\einsteinbinary_BRP4G_1.33_windows_x86_64__BRP4X64.exe caused a Breakpoint at location 17dad532 in module C:\WINDOWS\System32\KERNELBASE.dll.
Call stack:
17DAD532 C:\WINDOWS\System32\KERNELBASE.dll:17DAD532 DebugBreak
0068C4E3 C:\BOINCdata\projects\einstein.phys.uwm.edu\einsteinbinary_BRP4G_1.33_windows_x86_64__BRP4X64.exe:0068C4E3[07:16:16][2104][WARN ] BOINC wants us to quit prematurely or we lost contact! Exiting...
</stderr_txt>
]]>
Keith Myers wrote: The
)
It's not an error really, it's just how BOINC reacted on task abortion via GUI.
I decided to abort that task cause it l ran more than 2 days of CPU time already and clearly would miss deadline.
Regarding batch of smaller tasks, thanks, I'll take this in account.
Regarding BOINC crashes & CPU - well, BOINC mostly works fine issue with particular type of app not with BOINC. I did many tasks on this particular PC it worked on SETI before now purely on E@h.
Currenly I just disabled this kind of work for this type of hosts (to not waste CPU time missing deadlines).
But if some debugging of possible issues with BRP large is needed I could enable work again and watch closely.
Keith Myers
)
Thanks for the link, actually I completely forgot about that issue. But, looking at the thread, it seems Richard gave some advice, I did as he proposed and issue was solved... Since then host works fine... Perhaps it's smth another...
For completeness: currently host is running BOINC 7.16.5 (x64)
Yes, I would say it isn't
)
Yes, I would say it isn't worth it on that cpu. I was amazed at how long the tasks take on that cpu.
I run both the BRP4 and BRP4G on my Jetson Nano and I do the BRP4 in ten minutes and the 4G in 1.4 hours.
No problem finishing them in time.
I suspect you will have more troubleshooting opportunities in a month or two down the road when we eventually get the new BRP7 tasks and applications.
Keith Myers wrote: I
)
Ok will attempt to keep an eye :)