Uploads seem to be stuck ATM

Forums › Cruncher's Corner

Richard Haselgrove
Richard Haselgrove
Joined: 10 Dec 05
Posts: 2143
Credit: 2961035978
RAC: 693083

The scheduler

7 Feb 2019 10:53:24 UTC
Message 169368
(moderation:
    )

    The scheduler responds

    07/02/2019 10:51:39 | Einstein@Home | Project is temporarily shut down for maintenance

    so it's probably deliberate.

    Richie
    Richie
    Joined: 7 Mar 14
    Posts: 656
    Credit: 1702989778
    RAC: 0

    Just a quick history save...

    12 Feb 2019 17:38:37 UTC
    Message 169465
    (moderation:
      )

      Just a quick history save... Uploads not working right now...

      HTTP transient error / 504 Gateway Time-out

      ... but I'm writing this here because this seems to be a copy of the already familiar thing... and the automatic procedure worked last time pretty fast already.

      edit: And uploads are working again.

      jay
      jay
      Joined: 25 Jan 07
      Posts: 99
      Credit: 84044023
      RAC: 0

      Today, 12 Feb. 2019,  uploads

      12 Feb 2019 17:45:07 UTC
      Message 169468
      (moderation:
        )

        Today, 12 Feb. 2019,  uploads fail with  ongoing retries.

        I set the  http_xfer_debug flag .

        Here is what the event log says:

        Tue 12 Feb 2019 12:31:36 PM EST |  | log flags: file_xfer, sched_ops, task, http_xfer_debug
        Tue 12 Feb 2019 12:32:01 PM EST | Einstein@Home | Started upload of LATeah1047L_108.0_0_0.0_3774134_1_1
        Tue 12 Feb 2019 12:33:02 PM EST |  | [http_xfer] [ID#124] HTTP: wrote 433 bytes
        Tue 12 Feb 2019 12:33:03 PM EST | Einstein@Home | Temporarily failed upload of LATeah1047L_108.0_0_0.0_3774134_1_1: transient HTTP error
        Tue 12 Feb 2019 12:33:03 PM EST | Einstein@Home | Backing off 00:47:06 on upload of LATeah1047L_108.0_0_0.0_3774134_1_1

         

        Is this a cold weather, network problem? a gremlin somewhere?

        Thanks!!

        Jay.

         

        Richard Haselgrove
        Richard Haselgrove
        Joined: 10 Dec 05
        Posts: 2143
        Credit: 2961035978
        RAC: 693083

        You need to use the

        12 Feb 2019 17:51:44 UTC
        Message 169469 in response to message 169468
        (moderation:
          )

          You need to use the http_debug flag (not xfer) to see the 504 gateway timeout problem.

          They don't yet know what's causing it, just relying on the scheduled restart to keep it under control for now. And it's worked - mine have cleared (again).

          Keith Myers
          Keith Myers
          Joined: 11 Feb 11
          Posts: 4970
          Credit: 18772377099
          RAC: 7213057

          Uploads are stalling now from

          16 Feb 2019 23:42:17 UTC
          Message 169578
          (moderation:
            )

            Uploads are stalling now from several machines.

             

            Gary Roberts
            Gary Roberts
            Moderator
            Joined: 9 Feb 05
            Posts: 5872
            Credit: 117787141929
            RAC: 34703030

            This was the second time

            17 Feb 2019 0:54:04 UTC
            Message 169581
            (moderation:
              )

              This was the second time recently.  The first happened around 6 hours earlier.  Both have been resolved by the automatic restart procedure, it would seem.  At least that seems to be working in a fairly timely manner.

               

              Cheers,
              Gary.

              Betreger
              Betreger
              Joined: 25 Feb 05
              Posts: 992
              Credit: 1594102326
              RAC: 767228

              Yep

              17 Feb 2019 0:56:31 UTC
              Message 169582 in response to message 169581
              (moderation:
                )

                Yep

                Keith Myers
                Keith Myers
                Joined: 11 Feb 11
                Posts: 4970
                Credit: 18772377099
                RAC: 7213057

                I'm getting stuck uploads on

                8 Mar 2019 21:42:06 UTC
                Message 170003
                (moderation:
                  )

                  I'm getting stuck uploads on all my hosts now at 100% and multi hour backoffs.

                   

                  Keith Myers
                  Keith Myers
                  Joined: 11 Feb 11
                  Posts: 4970
                  Credit: 18772377099
                  RAC: 7213057

                  Still waiting on the

                  8 Mar 2019 22:29:43 UTC
                  Message 170005
                  (moderation:
                    )

                    Still waiting on the automatic restart procedure I guess.  They are preventing downloads/uploads on my other projects also.

                     

                    Richard Haselgrove
                    Richard Haselgrove
                    Joined: 10 Dec 05
                    Posts: 2143
                    Credit: 2961035978
                    RAC: 693083

                    Keith Myers wrote:They are

                    8 Mar 2019 22:34:52 UTC
                    Message 170006 in response to message 170005
                    (moderation:
                      )

                      Keith Myers wrote:
                      They are preventing downloads/uploads on my other projects also.

                      How? Message log entries?

                      Too many uploads will prevent work fetch from the same project, but shouldn't affect other projects.

                      Edit - this machine has 20 uploads queued for Einstein, but fetched from SETI while I typed.

                      Comment viewing options

                      Select your preferred way to display the comments and click "Save settings" to activate your changes.

                      Forums › Cruncher's Corner