arXiv moving from Cornell servers to Google Cloud • Hacker News Board

Re: arXiv moving from Cornell servers to Google Cloud

#41

Post by perihelions » Fri, Apr 18, 2025, 11:41 AM UTC

londons_explore wrote: post #20

dig1 wrote: post #18

> We are already underway on the arXiv CE ("Cloud Edition") project... replace the portion of our backends still written in perl and PHP...re-architect our article processing to be fully asynchronous,.. we can deploy via Kubernetes or services like Google Cloud Run...improve our monitoring and logging facilities Why do I smell someone from G was there and sold them fancy cloud story (or they wanted VMs and reseller s…

This.... I bet arXiv was run on server hardware costing under $10k before... And now it'll end up costing $10k per month (with free credit from Google which will eventually go away and then arXiv will shut down or be forced to go commercial)

arXiv budgets about $88,000/year for server costs as of 2019,

(pdf) https://info.arxiv.org/about/reports/arXiv_CY19_midyear.pdf

Re: arXiv moving from Cornell servers to Google Cloud

#42

Post by pharrington » Fri, Apr 18, 2025, 11:44 AM UTC

johann8384 wrote: post #35

einpoklum wrote: post #22

Another piece of (effectively) public infrastructure coming under the control of the mega-corporations, and Alphabet specifically.

I don't understand why people are being so dramatic about them using GCP.

Google absolutely will pull the rug on arXiv in the future. Any corporate entity would.

Re: arXiv moving from Cornell servers to Google Cloud

#43

Post by darkoob12 » Fri, Apr 18, 2025, 11:44 AM UTC

Fantastic. Now countries like Iran are going to be blocked. Internet is not a public network anymore It is owned by mostly American cooperation and they will decide what content to show and which group of people can access it.

Re: arXiv moving from Cornell servers to Google Cloud

#44

Post by gcr » Fri, Apr 18, 2025, 11:45 AM UTC

Cornell is currently in hiring freeze. These roles will not be filled.

Source: I applied to a Cornell-related lab in March. A week after submitting my application the role was rescinded and my contact emailed me explaining the situation.

Re: arXiv moving from Cornell servers to Google Cloud

#45

Post by harywilke » Fri, Apr 18, 2025, 11:47 AM UTC

This is something that started as far back as march 2023.

https://investinopen.org/blog/ioi-partners-with-arxiv-to-dev...

https://blog.arxiv.org/2023/06/12/arxiv-is-hiring-software/

Re: arXiv moving from Cornell servers to Google Cloud

#46

Post by jstummbillig » Fri, Apr 18, 2025, 11:48 AM UTC

dig1 wrote: post #18

> We are already underway on the arXiv CE ("Cloud Edition") project... replace the portion of our backends still written in perl and PHP...re-architect our article processing to be fully asynchronous,.. we can deploy via Kubernetes or services like Google Cloud Run...improve our monitoring and logging facilities Why do I smell someone from G was there and sold them fancy cloud story (or they wanted VMs and reseller s…

I noticed that while everyone on hn is quite clever, we are regularly not clever enough to assume that other people in similar settings are just as clever, and recognize when they probably spent a lot more time thinking about an issue we just skim the headline of.

Re: arXiv moving from Cornell servers to Google Cloud

#47

Post by specialp » Fri, Apr 18, 2025, 11:48 AM UTC

dig1 wrote: post #18

> We are already underway on the arXiv CE ("Cloud Edition") project... replace the portion of our backends still written in perl and PHP...re-architect our article processing to be fully asynchronous,.. we can deploy via Kubernetes or services like Google Cloud Run...improve our monitoring and logging facilities Why do I smell someone from G was there and sold them fancy cloud story (or they wanted VMs and reseller s…

I don't think it is that. I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling. As a primary source of information there is a lot of traffic. They do have technical issues from time to time due to this demand, and I think their stability is just due to the exceptional amount of effort they take to keep it going. They are also getting more submissions and interest.

Kubernetes does add complexity but it does add a lot of good things too. Auto scaling, cycling of unhealthy pods, and failover of failed nodes are some of them. I know there is this feeling here sometimes that cloud services and orchestrated containers are too much for many applications, but if you are running a very busy site like arXiv I can't see how running on bare metal is going to be better for your staff and experience. I don't think they are naive and got conned into GCP as the OP alludes to. They are smart people that are dealing with scaling and tech debt issues just like we all end up with at some point in our careers.

Re: arXiv moving from Cornell servers to Google Cloud

#48

Post by notpushkin » Fri, Apr 18, 2025, 11:52 AM UTC

sightbroke wrote: post #24

dig1 wrote: post #18

> We are already underway on the arXiv CE ("Cloud Edition") project... replace the portion of our backends still written in perl and PHP...re-architect our article processing to be fully asynchronous,.. we can deploy via Kubernetes or services like Google Cloud Run...improve our monitoring and logging facilities Why do I smell someone from G was there and sold them fancy cloud story (or they wanted VMs and reseller s…

> This is a project to re-home all arXiv services from VMs at Cornell to a cloud provider (Google Cloud). They are already using VMs but one of the things it'll do is: > containerize all, or nearly all arXiv services so we can deploy via Kubernetes or services like Google Cloud Run And further state: > The modernization will enable: - arXiv to expand the subject areas that we cover - improve the metadata we collect a…

Containers part I can understand. Why not spin up a tiny Docker Swarm (or k3s/k0s) cluster instead of straight out going to Google though?

Re: arXiv moving from Cornell servers to Google Cloud

#49

Post by whygcp » Fri, Apr 18, 2025, 11:54 AM UTC

darkoob12 wrote: post #43

Fantastic. Now countries like Iran are going to be blocked. Internet is not a public network anymore It is owned by mostly American cooperation and they will decide what content to show and which group of people can access it.

That's true. I recently had to move a VM from gcp to hetzner because gcp would silently drop all packets to some countries, Iran included. And a Stack overflow question was the easiest way to learn about it, not gcp docs.

Re: arXiv moving from Cornell servers to Google Cloud

#50

Post by CharlieDigital » Fri, Apr 18, 2025, 11:54 AM UTC

specialp wrote: post #47

dig1 wrote: post #18

> We are already underway on the arXiv CE ("Cloud Edition") project... replace the portion of our backends still written in perl and PHP...re-architect our article processing to be fully asynchronous,.. we can deploy via Kubernetes or services like Google Cloud Run...improve our monitoring and logging facilities Why do I smell someone from G was there and sold them fancy cloud story (or they wanted VMs and reseller s…

I don't think it is that. I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling. As a primary source of information there is a lot of traffic. They do have technical issues from time to time due to this demand, and I think their stability is just due to the exceptional amount of effort they take to keep it going. They are also getting more submissions and i…

Sound like all they needed was a CDN if the problem is AI crawlers. Adding auto-scaling compute just increases costs faster.