Brief project down-time TODAY

Submitted on 21 Oct 2016 8:21:15 UTC

Hi,

Due to an urgent security update we need to shut down the project today. The down-time shouldn't last longer than an hour so you shouldn't notice any significant impact on your crunching efforts. We're going to start as soon as we're ready and we'll post here when we're done.

Thanks,
Oliver 

Comments

Oliver Behnke
Oliver Behnke
Moderator
Administrator
Joined: 4 Sep 07
Posts: 987
Credit: 25171438
RAC: 4

Ok, 16 servers updated and

21 Oct 2016 14:30:23 UTC
Message 150953
(moderation:
    )

    Ok, 16 servers updated and rebooted. The project is fully up and running again.

    Thanks for your patience,
    Oliver 

    Einstein@Home Project

    cliff
    cliff
    Joined: 15 Feb 12
    Posts: 176
    Credit: 283452444
    RAC: 0

    There may be a problem, cant

    21 Oct 2016 14:43:14 UTC
    Message 150954
    (moderation:
      )

      There may be a problem, cant upload results for some reason:-/

      21/10/2016 15:35:15 | Einstein@Home | Temporarily failed upload of p2030.20141201.G193.69-00.79.C.b0s0g0.00000_2448_0_0: transient HTTP error

      Followed by

      21/10/2016 15:39:40 |  | Project communication failed: attempting access to reference site
      21/10/2016 15:39:42 |  | Internet access OK - project servers may be temporarily down.

      Guess summats not quite right..

      Regards,

       

      Cliff,

      Been there, Done that, Still no damm T Shirt.

      Todderbert
      Todderbert
      Joined: 3 Jun 15
      Posts: 1285
      Credit: 645963019
      RAC: 0

      I'm getting similar results

      21 Oct 2016 15:10:19 UTC
      Message 150955
      (moderation:
        )

        I'm getting similar results over here.

        10/21/2016 10:07:38 AM |  | Project communication failed: attempting access to reference site
        10/21/2016 10:07:38 AM | Einstein@Home | Backing off 00:03:38 on upload of p2030.20141124.G193.37+00.98.S.b1s0g0.00000_3008_0_14
        10/21/2016 10:07:40 AM |  | Internet access OK - project servers may be temporarily down.

        cliff
        cliff
        Joined: 15 Feb 12
        Posts: 176
        Credit: 283452444
        RAC: 0

        Yeah, its a problem, I have

        21 Oct 2016 16:25:42 UTC
        Message 150960 in response to message 150955
        (moderation:
          )

          Yeah, its a problem, I have more than 10 WU trying to upload:-(

          Last time I transient http errors it was SETI having network problems.. Not had this sorta thing with E@H though.

           

           

           

          Cliff,

          Been there, Done that, Still no damm T Shirt.

          Sir Rodney Ffing
          Sir Rodney Ffing
          Joined: 8 Nov 15
          Posts: 165
          Credit: 473102
          RAC: 0

          Same problem. Transfers tab

          21 Oct 2016 16:56:11 UTC
          Message 150962
          (moderation:
            )

            Same problem. Transfers tab is saying 6 hour backoff. Want to shut down the computer long before that.

            Mumak
            Joined: 26 Feb 13
            Posts: 335
            Credit: 3548148889
            RAC: 1239943

            Seems the staff has left for

            21 Oct 2016 17:21:31 UTC
            Message 150964
            (moderation:
              )

              Seems the staff has left for weekend.. This will be a busy Monday ;-)

              Richard Haselgrove
              Richard Haselgrove
              Joined: 10 Dec 05
              Posts: 2143
              Credit: 2974154666
              RAC: 798764

              BRP4G tasks are giving a

              21 Oct 2016 17:27:28 UTC
              Message 150966
              (moderation:
                )

                BRP4G tasks are giving a nasty http_debug error log when attempting uploads.

                
21/10/2016 18:22:13 | Einstein@Home | Started upload of p2030.20141201.G193.69-00.79.C.b2s0g0.00000_1440_0_1
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Info: Connection 1157 seems to be dead!
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Info: Closing connection 1157
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Info: Found bundle for host einstein3.aei.uni-hannover.de: 0x3eca770 [can pipeline]
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Info: Hostname einstein3.aei.uni-hannover.de was found in DNS cache
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Info: Trying 130.75.116.33...
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Info: Connected to einstein3.aei.uni-hannover.de (130.75.116.33) port 80 (#1159)
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Sent header to server: POST /EinsteinAtHome/cgi-bin/file_upload_handler_medium HTTP/1.1
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Sent header to server: Host: einstein3.aei.uni-hannover.de
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.7.0)
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Sent header to server: Accept: */*
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Sent header to server: Accept-Encoding: deflate, gzip
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Sent header to server: Content-Type: application/x-www-form-urlencoded
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Sent header to server: Accept-Language: en_GB
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Sent header to server: Content-Length: 4749
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Sent header to server: Expect: 100-continue
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Sent header to server:
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Received header from server: HTTP/1.1 100 Continue
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Info: We are completely uploaded and fine
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Received header from server: HTTP/1.1 404 Not Found
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Received header from server: Server: nginx/1.2.1
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Received header from server: Date: Fri, 21 Oct 2016 17:22:08 GMT
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Received header from server: Content-Type: text/html
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Received header from server: Content-Length: 168
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Received header from server: Connection: close
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Received header from server:
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Received header from server: <html>
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Received header from server: <head><title>404 Not Found</title></head>
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Received header from server: <body bgcolor="white">
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Received header from server: <center><h1>404 Not Found</h1></center>
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Received header from server: <hr><center>nginx/1.2.1</center>
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Received header from server: </body>
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Received header from server: </html>
21/10/2016 18:22:13 | Einstein@Home | [http] [ID#1259] Info: Closing connection 1159
21/10/2016 18:22:14 | Einstein@Home | Backing off 00:10:43 on upload of p2030.20141201.G193.69-00.79.C.b2s0g0.00000_1440_0_1

                HTTP/1.1 404 Not Found isn't a reassuring sign.

                cliff
                cliff
                Joined: 15 Feb 12
                Posts: 176
                Credit: 283452444
                RAC: 0

                Seems to me that this is

                21 Oct 2016 17:46:34 UTC
                Message 150968
                (moderation:
                  )

                  Seems to me that this is pretty standard for faults, either during the weekend/holiday or just prior, when there is no one to deal with said problem.. Murphy has struck once again:-(

                  Cheers

                  Cliff,

                  Been there, Done that, Still no damm T Shirt.

                  James L. Neill
                  James L. Neill
                  Joined: 14 Dec 10
                  Posts: 13
                  Credit: 141558696
                  RAC: 0

                  Cliff, you forgot about

                  21 Oct 2016 18:13:08 UTC
                  Message 150969
                  (moderation:
                    )

                    Cliff, you forgot about Murphy's little cousin called Sod! I cannot upload either

                     

                    Richard Haselgrove
                    Richard Haselgrove
                    Joined: 10 Dec 05
                    Posts: 2143
                    Credit: 2974154666
                    RAC: 798764

                    I think we might have

                    21 Oct 2016 19:25:47 UTC
                    Message 150972
                    (moderation:
                      )

                      I think we might have clearance on this - five machines have all just uploaded and reported.

                      Kudos to whoever stayed late and fixed it.

                      James L. Neill
                      James L. Neill
                      Joined: 14 Dec 10
                      Posts: 13
                      Credit: 141558696
                      RAC: 0

                      I would second Richard's

                      21 Oct 2016 19:44:52 UTC
                      Message 150974
                      (moderation:
                        )

                        I would second Richard's comment as my machine has just cleared. To whoever fixed things  a very big "Thank you"

                        morgan
                        morgan
                        Joined: 14 Dec 05
                        Posts: 2
                        Credit: 16981967
                        RAC: 0

                        Boinc-manager here says:

                        21 Oct 2016 19:58:43 UTC
                        Message 150975
                        (moderation:
                          )

                          Boinc-manager here says: -Project file upload handler is missing-

                          So no upload from me either!

                           

                           

                          mmonnin
                          mmonnin
                          Joined: 29 May 16
                          Posts: 292
                          Credit: 3444726540
                          RAC: 1187580

                          All my units have uploaded in

                          21 Oct 2016 20:06:58 UTC
                          Message 150976
                          (moderation:
                            )

                            All my units have uploaded in the past half hour or so after the down time.

                            morgan
                            morgan
                            Joined: 14 Dec 05
                            Posts: 2
                            Credit: 16981967
                            RAC: 0

                            After a forced retry(upload

                            21 Oct 2016 20:11:47 UTC
                            Message 150977
                            (moderation:
                              )

                              After a forced retry(upload ,all seem`s to be working OK here again Smile

                              Thanks 

                              Oliver Behnke
                              Oliver Behnke
                              Moderator
                              Administrator
                              Joined: 4 Sep 07
                              Posts: 987
                              Credit: 25171438
                              RAC: 4

                              See  Let us know if you still

                              24 Oct 2016 8:19:00 UTC
                              Message 151024
                              (moderation:
                                )

                                We did of course check the upload servers after their reboot and they worked just fine. They might have been overloaded at first or your clients might have went into their exponential back-off. Let us know if you still got stuck uploads...

                                Oliver

                                Einstein@Home Project

                                den777
                                den777
                                Joined: 29 May 13
                                Posts: 9
                                Credit: 57622282
                                RAC: 0

                                >urgent security

                                25 Oct 2016 10:56:18 UTC
                                Message 151045
                                (moderation:
                                  )

                                  >urgent security update

                                  Dirty Cow vulnerabilty?  

                                  Oliver Behnke
                                  Oliver Behnke
                                  Moderator
                                  Administrator
                                  Joined: 4 Sep 07
                                  Posts: 987
                                  Credit: 25171438
                                  RAC: 4

                                  Maybe 

                                  25 Oct 2016 11:02:09 UTC
                                  Message 151046
                                  (moderation:
                                    )

                                    Maybe Wink

                                    Einstein@Home Project