Live data from Hacker News

arXiv moving from Cornell servers to Google Cloud

info.arxiv.org

101–110 of 178 posts

Re: arXiv moving from Cornell servers to Google Cloud

#101
post #18

> We are already underway on the arXiv CE ("Cloud Edition") project... replace the portion of our backends still written in perl and PHP...re-architect our article processing to be fully asynchronous,.. we can deploy via Kubernetes or services like Google Cloud Run...improve our monitoring and logging facilities Why do I smell someone from G was there and sold them fancy cloud story (or they wanted VMs and reseller s…

I don't think it is that. I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling. As a primary source of information there is a lot of traffic. They do have technical issues from time to time due to this demand, and I think their stability is just due to the exceptional amount of effort they take to keep it going. They are also getting more submissions and i…

GC seems like a bad choice to me. The GC CLI and tools aren’t terrible, but their services in-general have historically been not been friendly to use and their support involves average documentation and humans that talk at users instead of serve them. Google as a company is not what it was, either. A lot of their funding was driven from advertising, and between social media, streaming, and LLMs, that seems like it may be starting to dry up.

Azure? Microsoft as a company is still the choice of most IT departments due to its ubiquitousness and low cost barrier to entry. I personally wouldn’t use Azure if I had the choice, because it’s easy and cheap at the surface, with hell underneath (except for products based on other things, like AD which was just a nice LDAP server, or C# which was modelled after Java).

I’d have gone with AWS. EKS isn’t bad to setup and is solid once it’s up. As far as the health of Amazon itself, China entering the their space hasn’t significantly changed their retail business, though eventually they’ll likely be in price wars.

The greatest risk to any cloud provider I think would be a war that could force national network isolation or government taking over infrastructure. And the grid would go down for a good while and water would stop, so everyone would start migrating, drinking polluted water, then maybe stealing food. At that point, whether or not they chose GC doesn’t matter anymore.

Re: arXiv moving from Cornell servers to Google Cloud

#102
post #96

Earlier quoted context omitted.

The people on both ends of that conversation (google vs cornell) are clever but the result will probably be enshittification.

I'm torn on flagging comments that throw out "enshittification". Do you feel that this stands in for actual thought?

[flagged]

Re: arXiv moving from Cornell servers to Google Cloud

#103
post #13
post #3

Job opening for the rare perl+latex hackers.

Perl is becoming rare, but is LaTeX also falling out of use for technical papers? Most of the scientific and CS papers I’ve seen lately still seem to use it, even those coming from Microsoft. That said, it's often generated via Org-mode or WYSIWYG tools these days.

> is LaTeX also falling out of use for technical papers?

In mechanical and aerospace engineering broadly, I see less use of LaTeX over time. It's hard to estimate precisely which percent of the market is LaTeX vs. Word vs. something else, but I think I can see trends. Almost no one where I work uses LaTeX, though LaTeX used to be more popular there.

I think it probably varies a lot by narrow specialty and publication venue too. Papers submitted to Journal of Fluid Mechanics seem to overwhelmingly use LaTeX. The main conference I would submit papers to during my PhD is primarily Word (though I used LaTeX). I have seen at least one Word-only engineering journal, though it wasn't something I would publish in.

Re: arXiv moving from Cornell servers to Google Cloud

#104
post #76

Earlier quoted context omitted.

Could have something to do with Cloudflare’s abhorrent sales practices.

Can you tell me more? I think my business needs some abhorrent sales practices. That's how it's done, right?

One example

https://robindev.substack.com/p/cloudflare-took-down-our-web...

Re: arXiv moving from Cornell servers to Google Cloud

#105
post #18

> We are already underway on the arXiv CE ("Cloud Edition") project... replace the portion of our backends still written in perl and PHP...re-architect our article processing to be fully asynchronous,.. we can deploy via Kubernetes or services like Google Cloud Run...improve our monitoring and logging facilities Why do I smell someone from G was there and sold them fancy cloud story (or they wanted VMs and reseller s…

I noticed that while everyone on hn is quite clever, we are regularly not clever enough to assume that other people in similar settings are just as clever, and recognize when they probably spent a lot more time thinking about an issue we just skim the headline of.

I’m not sure of the perspective of the OP but his comment hits home in that a common theme for the past ten years of my career has been “let’s move something to X, Y, and Z because that’s what Google says you should do.”

Note that Google doesn’t outright define an architecture for anyone, but people who worked at Google who come in as the hot hire bring that with them. Ditto for other large employers of the day. One of my mentors had to deal with this when Yahoo was the big deal.

In some cases, when abstractions are otherwise correct, this hasn’t been a big deal for the software projects and companies I’ve been involved with. It’s simply “there’s an off the shelf well supported industry standard we can use so we can focus on our customer/end goal/value add/etc.” Using an alternative docker runtime “that Google recommends” (aka is suggested by Kubernetes) is just a choice.

Where people get bit and then look at this with a squint, is when you work at several places where on the suggestion of the new big hire from Google/Amazon/Meta/etc, software that runs just fine on a couple server instances and has a low and stable KTLO budget ends up being broken down into microservices on Kubernetes and suddenly needs a team of three and doesn’t provide any additional value.

The worst I’ve experienced is a company that ended up listing the cost of maintaining their fork of Kubernetes and development environment as a material reason for a layoff.

Google’s marketing arm also has made deals to help move people to Google Cloud from AWS. Where I am working now this didn’t work to plan on either side it seems so we’re staying dual cloud, a situation no one is happy about. Before my time there was an executive on the finance side that believed Google was always the company to bet on and didn’t see Amazon as more than a book store. Also money. Different type of hubris, different type of pressure, same technical outcome as a CTO that runs on a “well Google says” game plan.

At the end of the day, Google is a big place and employs a lot of people. You’re going to have a lot of individuals who experience hucksters trying to parlay Google experience into an executive or high ranked IC role and they’re going to lean on what they know. That has nothing to do with Google itself, but their attempts to pry people away from AWS are about the same flavor from my personal experience.

Re: arXiv moving from Cornell servers to Google Cloud

#106
post #104

Earlier quoted context omitted.

Can you tell me more? I think my business needs some abhorrent sales practices. That's how it's done, right?

One example https://robindev.substack.com/p/cloudflare-took-down-our-web...

Wow, okay. That's a little too extreme. How is cloudflare acting insecure when so large? hmmm, confused.

Re: arXiv moving from Cornell servers to Google Cloud

#107
post #59

Earlier quoted context omitted.

I don't think it is that. I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling. As a primary source of information there is a lot of traffic. They do have technical issues from time to time due to this demand, and I think their stability is just due to the exceptional amount of effort they take to keep it going. They are also getting more submissions and i…

> I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling Funny, I also work on academic sites (much smaller than arXiv) and we're looking at moving from AWS to bare metal for the same reason. The $90/TB AWS bandwidth exit tariff can be a budget killer if people write custom scripts to download all your stuff; better to slow down than 10x the monthly budget.…

The two are not comparable. The 1TB of transit at Amazon can be subdivided over many recipients, while the solid state drive is empty and only can be sent to one.

That said, I agree that transit costs are too high.

Re: arXiv moving from Cornell servers to Google Cloud

#108
post #18

> We are already underway on the arXiv CE ("Cloud Edition") project... replace the portion of our backends still written in perl and PHP...re-architect our article processing to be fully asynchronous,.. we can deploy via Kubernetes or services like Google Cloud Run...improve our monitoring and logging facilities Why do I smell someone from G was there and sold them fancy cloud story (or they wanted VMs and reseller s…

I noticed that while everyone on hn is quite clever, we are regularly not clever enough to assume that other people in similar settings are just as clever, and recognize when they probably spent a lot more time thinking about an issue we just skim the headline of.

> I noticed that while everyone on hn is quite clever

Are we though? I see a pockets of world-expert level knowledge, some reasonable shop talk, and quite a bit of really dumb nonsense that is contradicted by my professional experience. Or just pedestrian level wrong. I mostly shitpost.

I don't have an opinion about arxiv's hosting, but it does read to be one of those projects that includes cleaning up of long standing technical debt that they probably couldn't get funded if not for the flashy goal. The flashy goal will, regardless of it's own merits, also be credited for improvements that they should have made all along.

Post reply on HN