Project files stuck in download

Forums › Problems and Bug Reports

Richard Haselgrove
Richard Haselgrove
Joined: 10 Dec 05
Posts: 2143
Credit: 3058316249
RAC: 2094571

ES-Six wrote: I tryed on a

17 Feb 2022 14:41:09 UTC
Message 192851 in response to message 192849
(moderation:
    )

    ES-Six wrote:

    I tryed on a linux computer and I encountered the same donwload errors but not on the same files (probably because as it's a linux computer, required files are not the same) :

    The program files are different, but the data files are the same. And it's data files that are missing.

    ES-Six wrote:

    Note : on both machines I added the project so it seem's to affect every newly attached hosts :)

    More likely, it's just that newly attached hosts are allocated newly generated data.

    ES-Six
    Joined: 6 Oct 18
    Posts: 18
    Credit: 30970261
    RAC: 1653

    Richard Haselgrove

    17 Feb 2022 14:50:07 UTC
    Message 192852 in response to message 192850
    (moderation:
      )

      Richard Haselgrove wrote:

      ES-Six wrote:

      ... an einstein@home server that is returning an HTTP error 404 when the BOINC client is attempting to download required files (specifically NVidia GPU and tasks related files).

      Have a nice day ...

      As you will see for yourself if you click on the links in my message 192810. Doesn't matter whether it's BOINC or a browser, Windows or Linux - if the file ain't there, it ain't there.

       

      Ok, I had some download errors for files with this kind of names on my windows machine, it confirm there is also tasks file download that is affected by the bug.

      I tested to download files throught my web browser too and got this same 404 error so yes, files are not there, but they should be there.

      I also tested to change the base domain to the following doamain : einsteinathome.org and it works for some files (binaries) but not for tasks files.

      San-Fernando-Valley
      San-Fernando-Valley
      Joined: 16 Mar 16
      Posts: 567
      Credit: 10962709465
      RAC: 11423943

      ES-Six wrote: For privacy

      17 Feb 2022 14:52:16 UTC
      Message 192853 in response to message 192848
      (moderation:
        )

        ES-Six wrote:

        For privacy reason, I hide all my computers from public users, so when you will finish to look at my specs I will hide them again.

        Sorry, I thought you had not noticed that the PCs are hidden. I'm finished.

        Strange errors.

        I have no further ideas - I'm just glad all of mine work!

        Cheers ....

        ES-Six
        Joined: 6 Oct 18
        Posts: 18
        Credit: 30970261
        RAC: 1653

        Richard Haselgrove

        17 Feb 2022 17:46:08 UTC
        Message 192854 in response to message 192851
        (moderation:
          )

          Richard Haselgrove wrote:

          ES-Six wrote:

          I tryed on a linux computer and I encountered the same donwload errors but not on the same files (probably because as it's a linux computer, required files are not the same) :

          The program files are different, but the data files are the same. And it's data files that are missing.

          ES-Six wrote:

          Note : on both machines I added the project so it seem's to affect every newly attached hosts :)

          More likely, it's just that newly attached hosts are allocated newly generated data.

          Yes program files are differents as the binary format is operating system and architecture related (x64, arm, etc.).

           

          I think to potentially 2 things that can explain this problem :

          1) A hard drive on one of the einstein@home server is failing and some files are unreadable

          2) A software update for boinc servers messed up something (e.g. bad generated URL in the project configuration) :

          - I also tryed with BOINC 7.16.16 and BOINC 7.16.20 and the problem appear for both versions of BOINC so it's not related to the BOINC clients. And last year, I used the same computers(and some more) to contribute without any issues.

          3) Workunit generation script is failing and outputing bad download path (but in this case it doesn't explain the 404 download error for binaries)

          4) Also a possibility but I don't think it's this one as I'm not alone to encounter download errors and I'm also not in the same country : network HTTP requests could be intercepted because they are not HTTPS so they are vulnerable to MITM attacks.  (eliminated, see below)

          5) Human error (unexpected file deletion on the server hard drive)

          Edit : I just used some proxies to test why files are not available and : I can eliminate the point 4) because the tested file is unreachable in all cases :

          tested file download link : http://einstein2.aei.uni-hannover.de/download/1c9/h1_0635.00_O3aC01Cl1In0

          ES-Six
          Joined: 6 Oct 18
          Posts: 18
          Credit: 30970261
          RAC: 1653

          San-Fernando-Valley

          17 Feb 2022 15:04:30 UTC
          Message 192855 in response to message 192853
          (moderation:
            )

            San-Fernando-Valley wrote:

            ES-Six wrote:

            For privacy reason, I hide all my computers from public users, so when you will finish to look at my specs I will hide them again.

            Sorry, I thought you had not noticed that the PCs are hidden. I'm finished.

            Strange errors.

            I have no further ideas - I'm just glad all of mine work!

            Cheers ....

            No worries,

            Computers are now hidden.

            Cheers :)

            ES-Six
            Joined: 6 Oct 18
            Posts: 18
            Credit: 30970261
            RAC: 1653

            I investigated more :I

            17 Feb 2022 18:27:06 UTC
            Message 192866
            (moderation:
              )

              I investigated more :

              I read more HTTP debug logs of my boinc clients and I observed links and domains used :

              I notices the client attempt to download files from two domains einstein-dl.syr.edu and einstein2.aei.uni-hannover.de domain.

              Exemple :

              http://einstein-dl.syr.edu/download/174/h1_0636.40_O3aC01Cl1In0

              and

              http://einstein2.aei.uni-hannover.de/download/8c/h1_0635.80_O3aC01Cl1In0

               

              So If I take a link not working because it returned 404 error and I change the domain like this :

              http://einstein2.aei.uni-hannover.de/download/1c9/h1_0635.00_O3aC01Cl1In0

              becomes

              http://einstein-dl.syr.edu/download/1c9/h1_0635.00_O3aC01Cl1In0

              I got a timeout error when trying to access file from a web browser.

              When I ping einstein-dl.syr.edu I got a timeout too.

              So : one of the einstein@home download server is down.

               

              That left us with these options :

              1) Einstein@home staff is making (unplanned) changes to the download servers (e.g. to replace the server) but it doesn't go as planned.

              2) A software update for boinc servers messed up something (e.g. bad generated URL in the project configuration) :

              - I also tryed with BOINC 7.16.16 and BOINC 7.16.20 and the problem appear for both versions of BOINC so it's not related to the BOINC clients. And last year, I used the same computers(and some more) to contribute without any issues.

              3) Workunit generation script is failing and outputing bad download path (but in this case it doesn't explain the 404 download error for binaries)

              4) Human error

               

              Server testing results with a buggy file download link :

              Server Respond to ping GET HTTP / succeed File download succeed (for a specific file)
              einstein-dl.syr.edu [timeout] (may be NOT intentional, server is down) NO [timeout] NO [timeout]
              einstein2.aei.uni-hannover.de [timeout] (may be intentional, server is up) YES [GOT redirection page to https://einsteinathome.org] NO [HTTP 404 error]

               

              I think there is one main server and one mirror server.

              In both cases, it's not a normal behaviour so there is definitely a problem on the einsteinathome.org server's side.

              Is there an admin here?

              Richard Haselgrove
              Richard Haselgrove
              Joined: 10 Dec 05
              Posts: 2143
              Credit: 3058316249
              RAC: 2094571

              I don't think is as simple as

              17 Feb 2022 19:34:02 UTC
              Message 192870 in response to message 192866
              (moderation:
                )

                I don't think is as simple as 'single server down'. I have several machines which are still downloading and processing GW tasks as normal - currently working on frequencies between 0627.20 and 0637.20 (the next group of figures after the h1_ prefix). I think that the problem tends to pounce when we've finished all the tasks for a given frequency band, and the project tries to send us the data files for a new frequency band. If they're not ready, or have gone awol - boom!

                With regard to the URLs: Einstein@Home has multiple servers around the world. In Europe, I get sent

                <file><br />
    <name>h1_0635.80_O3aC01Cl1In0</name><br />
    <nbytes>17582296.000000</nbytes><br />
    <max_nbytes>0.000000</max_nbytes><br />
    <md5_cksum>4f85b13047f9f20465a174337ff51311</md5_cksum><br />
    <status>1</status><br />
    <sticky/><br />
    <download_url>http://einstein2.aei.uni-hannover.de/download/8c/h1_0635.80_O3aC01Cl1In0</download_url><br />
    <download_url>http://einstein-dl.syr.edu/download/8c/h1_0635.80_O3aC01Cl1In0</download_url><br />
</file>

                with the European server (hannover.de) listed first. I think that people further west - in the Americas - get directed to (?) Syracuse University (syr.edu) in preference. The data files on both servers should be kept as exact mirrors of each other.

                The project administrators don't really have time to sit and read these messages. They usually rely on the forum moderators to alert them to developing problems. I had hoped that Gary Roberts might have picked up on this one: if he hasn't commented by tomorrow, I'll reach out to Bernd myself.

                mikey
                mikey
                Joined: 22 Jan 05
                Posts: 12958
                Credit: 1884513703
                RAC: 18024

                ES-Six wrote: I

                17 Feb 2022 23:17:31 UTC
                Message 192883 in response to message 192866
                (moderation:
                  )

                  ES-Six wrote:

                  I investigated more :

                  I read more HTTP debug logs of my boinc clients and I observed links and domains used :

                  I notices the client attempt to download files from two domains einstein-dl.syr.edu and einstein2.aei.uni-hannover.de domain. 

                  Do you use a VPN? If so try telling it you are in the US and see what happens, Try Canada, South America etc etc as since it works for some people you just need to make it connect to those Servers instead of the 2 that are failing for you.

                  ES-Six
                  Joined: 6 Oct 18
                  Posts: 18
                  Credit: 30970261
                  RAC: 1653

                  mikey wrote:ES-Six

                  18 Feb 2022 11:24:59 UTC
                  Message 192896 in response to message 192883
                  (moderation:
                    )

                    mikey wrote:

                    ES-Six wrote:

                    I investigated more :

                    I read more HTTP debug logs of my boinc clients and I observed links and domains used :

                    I notices the client attempt to download files from two domains einstein-dl.syr.edu and einstein2.aei.uni-hannover.de domain. 

                    Do you use a VPN? If so try telling it you are in the US and see what happens, Try Canada, South America etc etc as since it works for some people you just need to make it connect to those Servers instead of the 2 that are failing for you.

                    Personally I don't use a VPN.

                    But, that's interesting, I think you think einstein@home is doing a sort of load balancing based on geographic location of the user IP si I will try using a VPN and see what happens.

                    ES-Six
                    Joined: 6 Oct 18
                    Posts: 18
                    Credit: 30970261
                    RAC: 1653

                    Richard Haselgrove wrote: I

                    18 Feb 2022 11:28:22 UTC
                    Message 192897 in response to message 192870
                    (moderation:
                      )

                      Richard Haselgrove wrote:

                      I don't think is as simple as 'single server down'. I have several machines which are still downloading and processing GW tasks as normal - currently working on frequencies between 0627.20 and 0637.20 (the next group of figures after the h1_ prefix). I think that the problem tends to pounce when we've finished all the tasks for a given frequency band, and the project tries to send us the data files for a new frequency band. If they're not ready, or have gone awol - boom!

                      With regard to the URLs: Einstein@Home has multiple servers around the world. In Europe, I get sent

                      <file><br />
    <name>h1_0635.80_O3aC01Cl1In0</name><br />
    <nbytes>17582296.000000</nbytes><br />
    <max_nbytes>0.000000</max_nbytes><br />
    <md5_cksum>4f85b13047f9f20465a174337ff51311</md5_cksum><br />
    <status>1</status><br />
    <sticky/><br />
    <download_url>http://einstein2.aei.uni-hannover.de/download/8c/h1_0635.80_O3aC01Cl1In0</download_url><br />
    <download_url>http://einstein-dl.syr.edu/download/8c/h1_0635.80_O3aC01Cl1In0</download_url><br />
</file>

                      with the European server (hannover.de) listed first. I think that people further west - in the Americas - get directed to (?) Syracuse University (syr.edu) in preference. The data files on both servers should be kept as exact mirrors of each other.

                      The project administrators don't really have time to sit and read these messages. They usually rely on the forum moderators to alert them to developing problems. I had hoped that Gary Roberts might have picked up on this one: if he hasn't commented by tomorrow, I'll reach out to Bernd myself.

                      Ok, it makes sense, I will try with a VPN to see what happens.

                      Comment viewing options

                      Select your preferred way to display the comments and click "Save settings" to activate your changes.

                      Forums › Problems and Bug Reports