Unplanned project downtime 26 Aug 2022
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4313
Credit: 250757146
RAC: 34310
27 Aug 2022 11:15:37 UTC
Topic 228036
(moderation:
There was a campus-wide power outage yesterday afternoon, and the UPS that powers our network equipment stayed offline even after the power was restored, until some manual intervention this morning.
We'll continue to scan the systems for trouble caused by the outage, so far we didn't find more things broken than a few disks.
We'll do a detailed post-mortem analysis on Monday.
Sorry for the unplanned outage and the trouble it may have caused on your end.
BM
No problem Bernd, thank you
)
No problem Bernd, thank you for staying on top of it and keeping us informed. I know that working on the weekend wasn't what you expected to do, but thanks again for staying on top of it.
Many of us had an alternate plan for the "just in case" scenario, so it really didn't matter that much.
Proud member of the Old Farts Association
thanks Bernd. The only
)
thanks Bernd.
The only problem I see now is a lot of stuck uploads that wont go through. not sure if it's just because many people are trying to upload as well, or is there's a problem with the upload server.
_________________________________________________________________________
Upload error
)
Upload error log:
27/08/2022 13:18:59 | Einstein@Home | [http] [ID#14980] Info: connect to 130.75.116.34 port 80 failed: No route to host
27/08/2022 13:18:59 | Einstein@Home | [http] [ID#14980] Info: Failed to connect to einstein4.aei.uni-hannover.de port 80: No route to host
27/08/2022 13:18:59 | Einstein@Home | [http] [ID#14980] Info: Closing connection 5909
27/08/2022 13:18:59 | Einstein@Home | [http] HTTP error: Couldn't connect to server
27/08/2022 13:19:00 | Einstein@Home | Temporarily failed upload of LATeah3012L12220819_892.0_0_0.0_20553435_0_0: connect() failed
Uploading and downloading has
)
Uploading and downloading has stabilised for me at least, no more project requested back-offs. Thanks for the 'weekend servicing' Bernd.
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
I still have tasks that will
)
I still have tasks that will not upload.
36653 Einstein@Home 8/27/2022 6:01:57 AM Started upload of M22_1_cfbf00002_segment_4_dms_200_185_550000_0_0
36654 Einstein@Home 8/27/2022 6:01:57 AM Started upload of M22_1_cfbf00002_segment_4_dms_200_185_700000_0_0
36655 Einstein@Home 8/27/2022 6:01:57 AM Started upload of M22_1_cfbf00002_segment_4_dms_200_185_750000_0_0
36656 Einstein@Home 8/27/2022 6:01:57 AM Started upload of M22_1_cfbf00002_segment_4_dms_200_185_800000_0_0
36657 Einstein@Home 8/27/2022 6:01:57 AM Started upload of M22_1_cfbf00002_segment_4_dms_200_185_600000_0_0
36658 Einstein@Home 8/27/2022 6:02:19 AM Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_550000_0_0: connect() failed
36659 Einstein@Home 8/27/2022 6:02:19 AM Backing off 00:31:42 on upload of M22_1_cfbf00002_segment_4_dms_200_185_550000_0_0
36660 Einstein@Home 8/27/2022 6:02:19 AM Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_700000_0_0: connect() failed
36661 Einstein@Home 8/27/2022 6:02:19 AM Backing off 00:25:34 on upload of M22_1_cfbf00002_segment_4_dms_200_185_700000_0_0
36662 Einstein@Home 8/27/2022 6:02:19 AM Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_750000_0_0: connect() failed
36663 Einstein@Home 8/27/2022 6:02:19 AM Backing off 00:05:11 on upload of M22_1_cfbf00002_segment_4_dms_200_185_750000_0_0
36664 Einstein@Home 8/27/2022 6:02:19 AM Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_800000_0_0: connect() failed
36665 Einstein@Home 8/27/2022 6:02:19 AM Backing off 00:06:17 on upload of M22_1_cfbf00002_segment_4_dms_200_185_800000_0_0
36666 Einstein@Home 8/27/2022 6:02:19 AM Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_600000_0_0: connect() failed
36667 Einstein@Home 8/27/2022 6:02:19 AM Backing off 00:06:11 on upload of M22_1_cfbf00002_segment_4_dms_200_185_600000_0_0
36668 8/27/2022 6:02:20 AM Project communication failed: attempting access to reference site
36669 8/27/2022 6:02:22 AM Internet access OK - project servers may be temporarily down.
Reno, NV Team: SETI.USA
The uploads are failing for
)
The uploads are failing for the Meerkat tasks. All FGRB uploads have been cleared.
I have 116 tasks failures,
)
I have 116 tasks failures, all uploaded.
Tullio
Harri Liljeroos wrote: The
)
same, only BRP7 tasks stuck for me now.
_________________________________________________________________________
I am also unable to upload
)
I am also unable to upload BRP7s.
FWIW, I occasionally get
)
FWIW, I occasionally get more BRP7s to download. But when completed they still will not upload.
Reno, NV Team: SETI.USA