Live data from Hacker News

Yandex.Mail's successful migration from Oracle to Postgres [pdf]

pgcon.org

211–220 of 222 posts

Re: Yandex.Mail's successful migration from Oracle to Postgres [pdf]

#211
post #158
post #47

Earlier quoted context omitted.

Russian email hosting. What could go wrong?

Mail delivery delays can get huge, that's what's wrong with Yandex. Mail routinely takes 10-20 minutes to arrive.

Please, send us headers for one of such messages to support@mail.yandex.com. You can get them this way https://yandex.com/support/mail/faq.xml#mail-titles.

Re: Yandex.Mail's successful migration from Oracle to Postgres [pdf]

#212
post #201

Earlier quoted context omitted.

I worked for Yandex. AMA. Will not violate NDA, obviously.

Are you able to say anything about the scale of their hardware? It says "3x hardware" but I'd love to hear how big that ends up being, at either the Oracle or Postgresql size.

Hardware for Oracle and PostgreSQL are different. We used to have big boxes for Oracle with lots of memory, SSD disks and rigid economy of CPU (small number of very fast cores because Oracle is licenced per CPU cores). And that hardware was really efficiently used in consuming nearly all system resources (CPU, I/O, disk space and even network bandwidth).

For PostgreSQL we use commodity servers and right now they are not really efficiently balanced (i.e. on some hosts we don't have enough disk space but other resources are utilized ~30%). So we are working on improving that. But I can't tell you exact numbers of hosts for Oracle or PostgreSQL (which is actually ~x3 from Oracle hosts number :)).

Re: Yandex.Mail's successful migration from Oracle to Postgres [pdf]

#213

Earlier quoted context omitted.

You don't need to look there, at all. Just look at the US aggressive wars and subversive activities in this century, e.g. Iraq, Libya and it becomes clear that the value of human rights abroad from the US point of view is exactly 0.

Iraq was a disgrace but bringing up Libya means you are either ignorant or being disingenuous. Gaddafi had imminent plans to commit another cleansing of the opposition. It was always going to be a lose-lose situation for human rights. Doing nothing is not always the right answer.

By calling Iraq a disgrace you are effectively calling the 9/11 a micro-disgrace. That's how these two differ in the order of numbers of lives lost.

Re: Yandex.Mail's successful migration from Oracle to Postgres [pdf]

#214
post #200

Earlier quoted context omitted.

A place that wasn't a police state of sorts with increasingly draconian legislation. Opposite end would possibly be Swiss or Icelanfic jurisdiction. I'd never use an email service in Russia. I have an untrusted, Gmail account for availability purposes. Confidential mail still encrypted and preferrably sent through Swiss account. Thinking about switching untrusted one over to Fastmail. Still surveillance state but les…

I recall a story about Swiss company, Crypto AG: http://mediafilter.org/caq/cryptogate/

I've told the story many times. It can happen anywhere. It was an unscrupulous group of people in it solely for the money primarily selling to people like NSA. The target wasn't popular with anyone. Guy running it was offered some cash to sabotage the equipment of that target to help a major customer. "Happy to oblige!"

My endorsement of Switzerland is about legal jurisdiction, culture that respects privacy, and stability. Well-intentioned owners won't be forced to do the kinds of things we see in America or Russia. At worst, it will be something selective with a warrant. A nice foundation to build on.

Re: Yandex.Mail's successful migration from Oracle to Postgres [pdf]

#215
post #203
post #88

Earlier quoted context omitted.

SQL Server performs slightly better than PostgreSQL for analytics. This is mostly because of parallel queries which will still be more efficient on SQL Server when PostgreSQL 9.6 is released. On the other hand, PostgreSQL can take advantage of ZFS compression which is great for archiving cold data.

Minor note: ZFS compression is great for live data too -- lz4 compression has very low CPU overhead.

Yes and no. It depends on the use case, if you do a lot of updates, you do want a small recordsize(8kb) and if so, you do not benefit greatly from LZ4.

Re: Yandex.Mail's successful migration from Oracle to Postgres [pdf]

#216

Earlier quoted context omitted.

> Those two countries are similar in so many ways. Like invading and annexing their neighbours?

Exactly. Although to be fair to the US, their military is quite a bit more powerful, so they've already pacified their neighbours and instead invade countries half way around the globe. Russia can't stretch itself as much. But yeah, same principle.

So, US invaded and annexed someone in past 50 years? How's the weather in your version of reality?

Re: Yandex.Mail's successful migration from Oracle to Postgres [pdf]

#217
post #198
post #186

Earlier quoted context omitted.

it wouldn't have been 10 man years if they had less PL/SQL. I don't understand why people keep falling into that trap of putting complex logic in stored procedures.

No, you are not right. In video (Q&A section in the end) you can hear that rewriting stored logic took a couple of weeks of one developer. Maybe a couple of month with good tests. It took much more time to rewrite backends logic because of lots of legacy code in many applications without using abstraction libraries.

Oh, cool. Can you expand on that? What kind of logic had to change? I mean, SQL queries are SQL queries? Is that because you were using low level C Oracle drivers of some sort? Also if you could say a couple of words about the MySQL failure it would be rather helpful because I might have a friend who might be currently deciding on whether to migrate to MySQL or Postgres from Oracle.

Re: Yandex.Mail's successful migration from Oracle to Postgres [pdf]

#218
post #211
post #158

Earlier quoted context omitted.

Mail delivery delays can get huge, that's what's wrong with Yandex. Mail routinely takes 10-20 minutes to arrive.

Please, send us headers for one of such messages to support@mail.yandex.com. You can get them this way https://yandex.com/support/mail/faq.xml#mail-titles .

Thanks, will do.

Re: Yandex.Mail's successful migration from Oracle to Postgres [pdf]

#219
post #217
post #198

Earlier quoted context omitted.

No, you are not right. In video (Q&A section in the end) you can hear that rewriting stored logic took a couple of weeks of one developer. Maybe a couple of month with good tests. It took much more time to rewrite backends logic because of lots of legacy code in many applications without using abstraction libraries.

Oh, cool. Can you expand on that? What kind of logic had to change? I mean, SQL queries are SQL queries? Is that because you were using low level C Oracle drivers of some sort? Also if you could say a couple of words about the MySQL failure it would be rather helpful because I might have a friend who might be currently deciding on whether to migrate to MySQL or Postgres from Oracle.

We had several kinds of stored logic in Oracle: 1. for logical consistency of data (i.e. when you store new message you should increment counter for folder, thread, etc.). It seems correct to have such logic in the database rather than in the application. 2. for common code (i.e. moving a message from one folder to another may be done from web backend, mobile backend, imap backend or pop3 backend). This kind of logic is easier to have in one place than in all backends. 3. some business logic (i.e. finding thread for a new incoming message was done in PL/SQL).

First type is still in the database. Some part of type 2 was moved to our abstraction library (that is used by all applications). Logic of thrird type was moved to our applications and it greatly reduced stored code size.

Re: Yandex.Mail's successful migration from Oracle to Postgres [pdf]

#220
post #217
post #198

Earlier quoted context omitted.

No, you are not right. In video (Q&A section in the end) you can hear that rewriting stored logic took a couple of weeks of one developer. Maybe a couple of month with good tests. It took much more time to rewrite backends logic because of lots of legacy code in many applications without using abstraction libraries.

Oh, cool. Can you expand on that? What kind of logic had to change? I mean, SQL queries are SQL queries? Is that because you were using low level C Oracle drivers of some sort? Also if you could say a couple of words about the MySQL failure it would be rather helpful because I might have a friend who might be currently deciding on whether to migrate to MySQL or Postgres from Oracle.

About failed attempt of moving to MySQL. It was more than 10 years ago and I wasn't working at Yandex at that time but AFAIK the main problems were not around technical problems, they were organizational - not all members of the team agreed on moving to MySQL.
Post reply on HN