arXiv moving from Cornell servers to Google Cloud

Post by [deleted] » Fri, Apr 18, 2025, 12:38 PM UTC

CharlieDigital wrote: post #50

specialp wrote: post #47

Earlier quoted context omitted.

I don't think it is that. I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling. As a primary source of information there is a lot of traffic. They do have technical issues from time to time due to this demand, and I think their stability is just due to the exceptional amount of effort they take to keep it going. They are also getting more submissions and i…

Sound like all they needed was a CDN if the problem is AI crawlers. Adding auto-scaling compute just increases costs faster.

Post by miyuru » Fri, Apr 18, 2025, 12:43 PM UTC

CharlieDigital wrote: post #50

specialp wrote: post #47

Earlier quoted context omitted.

I don't think it is that. I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling. As a primary source of information there is a lot of traffic. They do have technical issues from time to time due to this demand, and I think their stability is just due to the exceptional amount of effort they take to keep it going. They are also getting more submissions and i…

Sound like all they needed was a CDN if the problem is AI crawlers. Adding auto-scaling compute just increases costs faster.

Post by whatever1 » Fri, Apr 18, 2025, 12:43 PM UTC

johann8384 wrote: post #31

elif wrote: post #19

Earlier quoted context omitted.

I think this is more likely a move to prevent Cornell funding from getting tied up with dictations about what gets published. But in all likelihood someone was probably just like "we're tired of doing ops on 2 decades old stacks"

It sounds like they are just using GCP. It doesn't really change that.

Post by infecto » Fri, Apr 18, 2025, 12:45 PM UTC

darkoob12 wrote: post #43

Fantastic. Now countries like Iran are going to be blocked. Internet is not a public network anymore It is owned by mostly American cooperation and they will decide what content to show and which group of people can access it.

Post by GTP » Fri, Apr 18, 2025, 12:46 PM UTC

specialp wrote: post #56

CharlieDigital wrote: post #50

Earlier quoted context omitted.

Sound like all they needed was a CDN if the problem is AI crawlers. Adding auto-scaling compute just increases costs faster.

CDN is one part of strategy to deal with load. But it is not the only solution unless your site is exclusively static content. Their search, APIs, submission pipelines, duplicate detectors and a lot of other things are not going to be powered by CDNs.