Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Cat-And-Mouse Story of Implementing Anti-Spam for Mail.ru

highscalability.com

Post reply on HN
Print view View original

21–28 of 28 posts

Re: The Cat-And-Mouse Story of Implementing Anti-Spam for Mail.ru

#21

Post by dang »

Normal_gaussian wrote: post #20
dang wrote: post #19

Earlier quoted context omitted.

We do that because then we can mark the subthread off-topic, putting it much lower on the page. If we didn't detach it first, marking off-topic would only put it lower among its siblings. This originated as a technique for dealing with "topjacking", i.e. comments that are posted as a reply to the top comment, not because they're relevant replies but simply to be nearer the top of the page. Later, we realized it was u…

Those may only be mind numbing for you who have to go through it every day, I really enjoyed the first two pages. Couldn't imagine having to do the replies though - a suprisingly large number had been customised!

> I really enjoyed the first two pages

Ha! Well that made me laugh out loud.

Yes, we write all those comments by hand and often make a point of customizing them. I even plant little bits of entertainment in there sometimes (hopefully not at anyone's expense), on the theory that the hivemind enjoys tidbits of variety.

Re: The Cat-And-Mouse Story of Implementing Anti-Spam for Mail.ru

#22

Post by dmm »

dang wrote: post #13
dmm wrote: post #6

You suck! I ran an email server from my home for several years. I could get a static ip but my isp wouldn't set my rdns for some reason. I had both spf and dkim correctly configured but people like you eventually forced me to move to commercial hosting.

Personal attacks are not allowed on Hacker News. We ban accounts that do this, so please don't do it again, even at someone you feel is unskilled at setting up a mail server. We detached this subthread from https://news.ycombinator.com/item?id=12398943 and marked it off-topic.

Sorry dang, kazinator.

Re: The Cat-And-Mouse Story of Implementing Anti-Spam for Mail.ru

#23

Post by marcosdumay »

kazinator wrote: post #3

One powerful rule that I use in my Exim setup is to drop any SMTP connection that doesn't have forward and backward DNS that match. This catches a big majority of all spam, with hardly any false positive downside. No credible in SMTP forwarding host has broken DNS (by definition, practically). Lots of spamming machines do. Of course, not all, but this is a game of percentages and multiple stages.

Never thought about it (and I've been thinking about email a lot recently), but now that people mentioned it...

Every mail sever defaults should be to trust an encrypted connection with a valid MX host name. Failing that, it may choose to check either reverse DNS or SPF. I see no reason for preferring any of those. Not checking anything by default isn't a good idea nowadays.

Re: The Cat-And-Mouse Story of Implementing Anti-Spam for Mail.ru

#24

Post by r00fus »

treve wrote: post #5
kazinator wrote: post #3

One powerful rule that I use in my Exim setup is to drop any SMTP connection that doesn't have forward and backward DNS that match. This catches a big majority of all spam, with hardly any false positive downside. No credible in SMTP forwarding host has broken DNS (by definition, practically). Lots of spamming machines do. Of course, not all, but this is a game of percentages and multiple stages.

You'll definitely lose email from Amazon SES then. What about checking SPF records instead?

Is this a bad thing, if you're trying to avoid spam?

Re: The Cat-And-Mouse Story of Implementing Anti-Spam for Mail.ru

#25

Post by bigbes »

Robin_Message wrote: post #16

The spam bit sounds cool but the nosql db sounds odd. E.G. That tree implementation with batched rebalancing sounds downright scary, and I thought avl was considered the worst kind of balanced tree due to excessive overhead (an int per node to red-black's bit) .

> Our investigation revealed that in case of frequent insertions and deletions Tarantool initiated a complex process of tree rebalancing (all our indexes were of TREE type).

It's all about 1.5. New version (1.6) uses brand new bps-tree, not sg/avl-tree. It behaves better on all workloads.

AVLTree was "temporary" hack. Our implementation works better, for their needs. BTW - AVL is not bad, but it's hard to implement a good one (believe me :) ).

Re: The Cat-And-Mouse Story of Implementing Anti-Spam for Mail.ru

#26

Post by jlgaddis »

kazinator wrote: post #15
Nux wrote: post #9

Earlier quoted context omitted.

So, do you require the PTR and A records of the remote host to match?

I don't work with DNS at the protocol level; it's done via the Exim "verify = reverse_host_lookup". The documentation says: This condition ensures that a verified host name has been looked up from the IP address of the client host. (This may have happened already if the host name was needed for checking a host list, or if the host matched host_lookup.) Verification ensures that the host name obtained from a reverse D…

The general term for this is "Forward Confirmed reverse DNS" (FCrDNS). I do these checks (plus others) on my Postfix mail systems. I would assume most of the other (sane) MTAs can perform these checks as well.

Re: The Cat-And-Mouse Story of Implementing Anti-Spam for Mail.ru

#27

Post by Robin_Message »

bigbes wrote: post #25
Robin_Message wrote: post #16

The spam bit sounds cool but the nosql db sounds odd. E.G. That tree implementation with batched rebalancing sounds downright scary, and I thought avl was considered the worst kind of balanced tree due to excessive overhead (an int per node to red-black's bit) .

> Our investigation revealed that in case of frequent insertions and deletions Tarantool initiated a complex process of tree rebalancing (all our indexes were of TREE type). It's all about 1.5. New version (1.6) uses brand new bps-tree, not sg/avl-tree. It behaves better on all workloads. AVLTree was "temporary" hack. Our implementation works better, for their needs. BTW - AVL is not bad, but it's hard to implement a…

Nice. Yeah, AVL is definitely okay but it makes me happy to hear something better is in there :)

What sort of tree is this new bps-tree?

Re: The Cat-And-Mouse Story of Implementing Anti-Spam for Mail.ru

#28

Post by bigbes »

Robin_Message wrote: post #27
bigbes wrote: post #25

Earlier quoted context omitted.

> Our investigation revealed that in case of frequent insertions and deletions Tarantool initiated a complex process of tree rebalancing (all our indexes were of TREE type). It's all about 1.5. New version (1.6) uses brand new bps-tree, not sg/avl-tree. It behaves better on all workloads. AVLTree was "temporary" hack. Our implementation works better, for their needs. BTW - AVL is not bad, but it's hard to implement a…

Nice. Yeah, AVL is definitely okay but it makes me happy to hear something better is in there :) What sort of tree is this new bps-tree?

bps-tree means B+-tree: unique combination of B+ and B tree. You can read wiki or "The Ubiquitous B-Tree" whitepaper https://wwwold.cs.umd.edu/class/fall2002/cmsc818s/Readings/b... .

You may read code of it here: https://github.com/tarantool/tarantool/blob/1.6/src/lib/sala... . It's very thoroughly commented.

Post reply on HN