Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

arXiv moving from Cornell servers to Google Cloud

info.arxiv.org

Post reply on HN
Print view View original

71–80 of 178 posts

Re: arXiv moving from Cornell servers to Google Cloud

#71

Post by masklinn »

Funes- wrote: post #67
specialp wrote: post #47

Earlier quoted context omitted.

I don't think it is that. I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling. As a primary source of information there is a lot of traffic. They do have technical issues from time to time due to this demand, and I think their stability is just due to the exceptional amount of effort they take to keep it going. They are also getting more submissions and i…

>AI crawling Can you not reliably block crawlers in this day and age?

AI crawlers are a plague, they are intentionally badly behaved and designed to be hard to flag without nuking legit traffic. That’s why projects like nepenthes exist.

Re: arXiv moving from Cornell servers to Google Cloud

#72

Post by johnisgood »

dewey wrote: post #32
einpoklum wrote: post #22

Another piece of (effectively) public infrastructure coming under the control of the mega-corporations, and Alphabet specifically.

Just because you are hosting at a company doesn’t mean they are “controlling” it.

Can't they stop providing services at a whim? Can't they do whatever with the data?

Re: arXiv moving from Cornell servers to Google Cloud

#74

Post by fuckbrownpeople »

wordofx wrote: post #66
jstummbillig wrote: post #46

Earlier quoted context omitted.

I noticed that while everyone on hn is quite clever, we are regularly not clever enough to assume that other people in similar settings are just as clever, and recognize when they probably spent a lot more time thinking about an issue we just skim the headline of.

[flagged]

[dead]

Re: arXiv moving from Cornell servers to Google Cloud

#75

Post by Imustaskforhelp »

specialp wrote: post #56
CharlieDigital wrote: post #50

Earlier quoted context omitted.

Sound like all they needed was a CDN if the problem is AI crawlers. Adding auto-scaling compute just increases costs faster.

CDN is one part of strategy to deal with load. But it is not the only solution unless your site is exclusively static content. Their search, APIs, submission pipelines, duplicate detectors and a lot of other things are not going to be powered by CDNs.

pardon me but cloudflare workers seem better for this approach.

If we can get for the fact that we require javascript to run it, aside from that. Cloudflare workers is literally the best single thing to happen at least to me. With a single domain, I have done so many personal projects for problems I found interesting and I built so many projects for literally free, no Credit card. No worries whatsoever.

I might ditch writing other languages for server based like golang even though I like golang more just because cloudflare workers exists.

Re: arXiv moving from Cornell servers to Google Cloud

#76

Post by mcmcmc »

Imustaskforhelp wrote: post #70
JackC wrote: post #59

Earlier quoted context omitted.

> I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling Funny, I also work on academic sites (much smaller than arXiv) and we're looking at moving from AWS to bare metal for the same reason. The $90/TB AWS bandwidth exit tariff can be a budget killer if people write custom scripts to download all your stuff; better to slow down than 10x the monthly budget.…

I don't understand, why don't you use cloudflare? Don't they have an unlimited egress policy with R1? Its way more predictable in my opinion that you only pay per month a fixed amount to your storage, it can also help the fact that its on the edge so users would get it way faster than lets say going to bare metal (unless you are provisioning a multi server approach and I think you might be using kubernetes there and…

Could have something to do with Cloudflare’s abhorrent sales practices.

Re: arXiv moving from Cornell servers to Google Cloud

#77

Post by specialp »

Funes- wrote: post #67
specialp wrote: post #47

Earlier quoted context omitted.

I don't think it is that. I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling. As a primary source of information there is a lot of traffic. They do have technical issues from time to time due to this demand, and I think their stability is just due to the exceptional amount of effort they take to keep it going. They are also getting more submissions and i…

>AI crawling Can you not reliably block crawlers in this day and age?

You can to some degree with Cloudflare and other solutions. But, do you want to block them all? AI is a very useful tool for people to discover information and summarize results. Especially in scholarly publishing where one would have to previously search on dumb keywords, and have to read loads of abstracts to find the research that pertains to their interests. So by blocking AI crawlers and bots completely, you are shutting off what will probably end up being the primary way people use your resource not too long from now. arXiv is a hub of research, and their mission is to make that research freely available to the world.

Re: arXiv moving from Cornell servers to Google Cloud

#78

Post by kelnos »

notpushkin wrote: post #48
sightbroke wrote: post #24

Earlier quoted context omitted.

> This is a project to re-home all arXiv services from VMs at Cornell to a cloud provider (Google Cloud). They are already using VMs but one of the things it'll do is: > containerize all, or nearly all arXiv services so we can deploy via Kubernetes or services like Google Cloud Run And further state: > The modernization will enable: - arXiv to expand the subject areas that we cover - improve the metadata we collect a…

Containers part I can understand. Why not spin up a tiny Docker Swarm (or k3s/k0s) cluster instead of straight out going to Google though?

Because those other things require more maintenance effort to run.

Getting creative is often just a pain in the ass. Doing the standard things, walking the well-trod path, is generally easier to do, even if it may not be the cheapest or most hardware/software-efficient thing to do.

Re: arXiv moving from Cornell servers to Google Cloud

#79

Post by Imustaskforhelp »

Funes- wrote: post #67
specialp wrote: post #47

Earlier quoted context omitted.

I don't think it is that. I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling. As a primary source of information there is a lot of traffic. They do have technical issues from time to time due to this demand, and I think their stability is just due to the exceptional amount of effort they take to keep it going. They are also getting more submissions and i…

>AI crawling Can you not reliably block crawlers in this day and age?

Dude, I don't want to sound cloudflare advocate because my last 2 comments on this thread are just shilling cloudflare...

but I think cloudflare is the answer to this thing as well.. (Sorry if I am being annoying) (Cloudflare isn't sponsoring me, I just love their service so much)

Re: arXiv moving from Cornell servers to Google Cloud

#80

Post by Orygin »

whygcp wrote: post #49
darkoob12 wrote: post #43

Fantastic. Now countries like Iran are going to be blocked. Internet is not a public network anymore It is owned by mostly American cooperation and they will decide what content to show and which group of people can access it.

That's true. I recently had to move a VM from gcp to hetzner because gcp would silently drop all packets to some countries, Iran included. And a Stack overflow question was the easiest way to learn about it, not gcp docs.

I have looked at it recently and it seems Iran is blocking GCP, not the other way around. Not sure if Google keep a doc up to date with who blocks them.
Post reply on HN