Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

arXiv moving from Cornell servers to Google Cloud

info.arxiv.org

Post reply on HN
Print view View original

61–70 of 178 posts

Re: arXiv moving from Cornell servers to Google Cloud

#61

Post by coliveira »

specialp wrote: post #56
CharlieDigital wrote: post #50

Earlier quoted context omitted.

Sound like all they needed was a CDN if the problem is AI crawlers. Adding auto-scaling compute just increases costs faster.

CDN is one part of strategy to deal with load. But it is not the only solution unless your site is exclusively static content. Their search, APIs, submission pipelines, duplicate detectors and a lot of other things are not going to be powered by CDNs.

All these services can be throttled to deal with AI. I don't see this as a justification. The idea that a service like arXiv should be run as a startup is, simply put, foolish.

Re: arXiv moving from Cornell servers to Google Cloud

#62

Post by moralestapia »

dig1 wrote: post #18

> We are already underway on the arXiv CE ("Cloud Edition") project... replace the portion of our backends still written in perl and PHP...re-architect our article processing to be fully asynchronous,.. we can deploy via Kubernetes or services like Google Cloud Run...improve our monitoring and logging facilities Why do I smell someone from G was there and sold them fancy cloud story (or they wanted VMs and reseller s…

>we can deploy via Kubernetes

Oh noes ... they got scammed

Re: arXiv moving from Cornell servers to Google Cloud

#63

Post by quantumHazer »

gcr wrote: post #44

Cornell is currently in hiring freeze. These roles will not be filled. Source: I applied to a Cornell-related lab in March. A week after submitting my application the role was rescinded and my contact emailed me explaining the situation.

Is it related with policies from the US administration?

Re: arXiv moving from Cornell servers to Google Cloud

#64

Post by bluenose69 »

rahen wrote: post #13
jeffbee wrote: post #3

Job opening for the rare perl+latex hackers.

Perl is becoming rare, but is LaTeX also falling out of use for technical papers? Most of the scientific and CS papers I’ve seen lately still seem to use it, even those coming from Microsoft. That said, it's often generated via Org-mode or WYSIWYG tools these days.

LaTeX is still very much in favour in my field of geophysical research. I believe that the same is true across most fields that rely on mathematical notation to explain things in written form. I've never found it difficult to write in a non-WSIWYG system ... indeed, it is really quite convenient.

Years ago I had a student in my class who was unable to make out written material, but had a machine that read text aloud. My class notes (some 300+ pages) are chock full of mathematics. I went to the student's house to see how that reading machine worked. Provided with LaTeX input, it said a lot of things like "backslash alpha" and "begin open brace equation close brace" stuff. I wrote a quick perl script to change it, so it said "alpha" and "begin equation". Presto -- it was exactly what the student needed. This was, as I say, many years ago. Maybe now there is software that can handle MSword files, etc., but that definitely did not exist at the time. The result? The student was able to take the class, and did very well in it.

Re: arXiv moving from Cornell servers to Google Cloud

#66

Post by wordofx »

jstummbillig wrote: post #46
dig1 wrote: post #18

> We are already underway on the arXiv CE ("Cloud Edition") project... replace the portion of our backends still written in perl and PHP...re-architect our article processing to be fully asynchronous,.. we can deploy via Kubernetes or services like Google Cloud Run...improve our monitoring and logging facilities Why do I smell someone from G was there and sold them fancy cloud story (or they wanted VMs and reseller s…

I noticed that while everyone on hn is quite clever, we are regularly not clever enough to assume that other people in similar settings are just as clever, and recognize when they probably spent a lot more time thinking about an issue we just skim the headline of.

[flagged]

Re: arXiv moving from Cornell servers to Google Cloud

#67

Post by Funes- »

specialp wrote: post #47
dig1 wrote: post #18

> We are already underway on the arXiv CE ("Cloud Edition") project... replace the portion of our backends still written in perl and PHP...re-architect our article processing to be fully asynchronous,.. we can deploy via Kubernetes or services like Google Cloud Run...improve our monitoring and logging facilities Why do I smell someone from G was there and sold them fancy cloud story (or they wanted VMs and reseller s…

I don't think it is that. I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling. As a primary source of information there is a lot of traffic. They do have technical issues from time to time due to this demand, and I think their stability is just due to the exceptional amount of effort they take to keep it going. They are also getting more submissions and i…

>AI crawling

Can you not reliably block crawlers in this day and age?

Re: arXiv moving from Cornell servers to Google Cloud

#68

Post by kadushka »

jstummbillig wrote: post #46
dig1 wrote: post #18

> We are already underway on the arXiv CE ("Cloud Edition") project... replace the portion of our backends still written in perl and PHP...re-architect our article processing to be fully asynchronous,.. we can deploy via Kubernetes or services like Google Cloud Run...improve our monitoring and logging facilities Why do I smell someone from G was there and sold them fancy cloud story (or they wanted VMs and reseller s…

I noticed that while everyone on hn is quite clever, we are regularly not clever enough to assume that other people in similar settings are just as clever, and recognize when they probably spent a lot more time thinking about an issue we just skim the headline of.

The people on both ends of that conversation (google vs cornell) are clever but the result will probably be enshittification.

Re: arXiv moving from Cornell servers to Google Cloud

#70

Post by Imustaskforhelp »

JackC wrote: post #59
specialp wrote: post #47

Earlier quoted context omitted.

I don't think it is that. I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling. As a primary source of information there is a lot of traffic. They do have technical issues from time to time due to this demand, and I think their stability is just due to the exceptional amount of effort they take to keep it going. They are also getting more submissions and i…

> I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling Funny, I also work on academic sites (much smaller than arXiv) and we're looking at moving from AWS to bare metal for the same reason. The $90/TB AWS bandwidth exit tariff can be a budget killer if people write custom scripts to download all your stuff; better to slow down than 10x the monthly budget.…

I don't understand, why don't you use cloudflare? Don't they have an unlimited egress policy with R1?

Its way more predictable in my opinion that you only pay per month a fixed amount to your storage, it can also help the fact that its on the edge so users would get it way faster than lets say going to bare metal (unless you are provisioning a multi server approach and I think you might be using kubernetes there and it might be a mess to handle I guess?)

Post reply on HN