Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

arXiv moving from Cornell servers to Google Cloud

info.arxiv.org

Post reply on HN
Print view View original

51–60 of 178 posts

Re: arXiv moving from Cornell servers to Google Cloud

#51

Post by bustling-noose »

My theory : Engineers rented servers and maintained them with software and packages and infrastructure scripts / code etc. Then this moved to cloud VMs cause it became easier for higher availability and sometimes cheaper too. Then VM costs started rising and cloud providers started offering tempting prices to use some of their services instead that integrated well with their VM infrastructure. Simultaneously engineers started to cost more money to maintain these systems and more people trained in 'cloud' became available for relatively cheap. So people moved their infrastructure to cloud offerings. Now both engineers and cloud services cost a lot of money. But now engineers who can maintain such infrastructure are far and few in between and also cost a lot of money while cloud offerings became turnkey literally.

So costs went from $80k an engineer a year maybe a decade ago with a few thousand to servers to $200k an engineer which you would struggle to find or $100k for a 'cloud engineer / architect' plus $100k to a cloud provider.

This sounds great in theory. Except that cloud providers are messy and once vendor locked in, you are in a big spiral. Secondly the costs can be hidden and climb exponentially if you don't know exactly what you are doing. You might also get into weird bugs that could be solved by a patch over a Monday to some package you could just update which might take months or never happen on a cloud provider. The reality of moving to cloud is not as rosy as it sounds.

Universities used to be the birth place of big projects that were created to solve problems they ran into hosting / running their own infrastructure. I hope that is still true.

Re: arXiv moving from Cornell servers to Google Cloud

#52

Post by [deleted] »

perihelions wrote: post #41
londons_explore wrote: post #20

Earlier quoted context omitted.

This.... I bet arXiv was run on server hardware costing under $10k before... And now it'll end up costing $10k per month (with free credit from Google which will eventually go away and then arXiv will shut down or be forced to go commercial)

arXiv budgets about $88,000/year for server costs as of 2019, (pdf) https://info.arxiv.org/about/reports/arXiv_CY19_midyear.pdf

[deleted]

Re: arXiv moving from Cornell servers to Google Cloud

#54

Post by elif »

johann8384 wrote: post #31
elif wrote: post #19

Earlier quoted context omitted.

I think this is more likely a move to prevent Cornell funding from getting tied up with dictations about what gets published. But in all likelihood someone was probably just like "we're tired of doing ops on 2 decades old stacks"

It sounds like they are just using GCP. It doesn't really change that.

Yea but GCP was state of the art 20 years ago, php+perl was already crufty

Re: arXiv moving from Cornell servers to Google Cloud

#55

Post by throwaway984393 »

johann8384 wrote: post #35
einpoklum wrote: post #22

Another piece of (effectively) public infrastructure coming under the control of the mega-corporations, and Alphabet specifically.

I don't understand why people are being so dramatic about them using GCP.

[dead]

Re: arXiv moving from Cornell servers to Google Cloud

#56

Post by specialp »

CharlieDigital wrote: post #50
specialp wrote: post #47

Earlier quoted context omitted.

I don't think it is that. I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling. As a primary source of information there is a lot of traffic. They do have technical issues from time to time due to this demand, and I think their stability is just due to the exceptional amount of effort they take to keep it going. They are also getting more submissions and i…

Sound like all they needed was a CDN if the problem is AI crawlers. Adding auto-scaling compute just increases costs faster.

CDN is one part of strategy to deal with load. But it is not the only solution unless your site is exclusively static content. Their search, APIs, submission pipelines, duplicate detectors and a lot of other things are not going to be powered by CDNs.

Re: arXiv moving from Cornell servers to Google Cloud

#58

Post by motorest »

specialp wrote: post #56
CharlieDigital wrote: post #50

Earlier quoted context omitted.

Sound like all they needed was a CDN if the problem is AI crawlers. Adding auto-scaling compute just increases costs faster.

CDN is one part of strategy to deal with load. But it is not the only solution unless your site is exclusively static content. Their search, APIs, submission pipelines, duplicate detectors and a lot of other things are not going to be powered by CDNs.

Thank you for the insight. It's very easy to prescribe simple solutions when we are oblivious to the actual problems being solved.

Re: arXiv moving from Cornell servers to Google Cloud

#59

Post by JackC »

specialp wrote: post #47
dig1 wrote: post #18

> We are already underway on the arXiv CE ("Cloud Edition") project... replace the portion of our backends still written in perl and PHP...re-architect our article processing to be fully asynchronous,.. we can deploy via Kubernetes or services like Google Cloud Run...improve our monitoring and logging facilities Why do I smell someone from G was there and sold them fancy cloud story (or they wanted VMs and reseller s…

I don't think it is that. I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling. As a primary source of information there is a lot of traffic. They do have technical issues from time to time due to this demand, and I think their stability is just due to the exceptional amount of effort they take to keep it going. They are also getting more submissions and i…

> I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling

Funny, I also work on academic sites (much smaller than arXiv) and we're looking at moving from AWS to bare metal for the same reason. The $90/TB AWS bandwidth exit tariff can be a budget killer if people write custom scripts to download all your stuff; better to slow down than 10x the monthly budget.

(I never thought about it this way, but Amazon charges less to same-day deliver a 1TB SSD drive for you to keep than it does to download a TB from AWS.)

Re: arXiv moving from Cornell servers to Google Cloud

#60

Post by orochimaaru »

progbits wrote: post #2

I'm sure many are going to hate on this, but a simple k8s setup (just container+deployment+ingress) is going to be good for preventing vendor lock in. Too bad it's US only, devops is actually a good role for full remote setups. If you can't make it work asynchronously across timezones you are doing it wrong :)

Hiring and payroll can be a complete nightmare especially with publicly funded organizations and initiatives like arxiv.

Also plenty of North Korean bad actors masquerading as US remote workers. I’m betting as European remote workers too. I like remote work but shit like that is why we can’t have good things.

https://www.yahoo.com/news/thousands-north-korean-workers-in...

Post reply on HN