Live data from Hacker News

Call me maybe: Aerospike

aphyr.com

51–60 of 132 posts

Re: Call me maybe: Aerospike

#51
post #9

The rotated A in the Aerospike logo reminds me of a system that's fallen over, and now you can't unsee it: http://www.aerospike.com/

"A system"? Not sure what you mean by that. To me, it was immediately apparent that they are making the whole word Aerospike look like a rocket, a motif they repeat through their home-page.

Worker 1: "The server has fallen over."

Worker 2: "I'll go restart it."

See also: The frequent interchangeability of the words "system" and "computer".

Re: Call me maybe: Aerospike

#52
post #16

I currently work for a company that uses Aerospike quite heavily. In the past couple weeks, we have begun to notice data inconsistencies in our counters. We are seeing fluctuations in the data, despite having no decrement operations. We have the enterprise edition of Aerospike, allowing us to be in constant contact with their support team and developers. A couple weeks later, and we still have no idea why this is hap…

What are you migrating to ?

As is away from Aerospike to... ?

Thanks, very interesting anecdote/case.

Re: Call me maybe: Aerospike

#53
post #32
post #24

Earlier quoted context omitted.

Very strange. I've never encountered something like that with any database or cache. Wonder if it's somehow related to the way that your cluster is setup? How big is this cluster? Are you writing and reading to the entire cluster, or do you have certain nodes that you write to and others that you read from?

I have to admit, I'm not 100% on the entire configuration. However... We have two clusters of 8 nodes each. Each cluster is setup with 2 factor replication. The clusters are setup with cross datacenter replication. Your read / write question is a little hard to answer. In Aerospike, a given key will always reside on the same node, something to do with how they optimize their storage. Which means that anytime you writ…

When Aerospike ships XDR batches it does not replay events, it just re-syncs the data. This is true even for increments. So if cluster A has 10 increments of n to n+10, and cluster B has 20 increments of n to n+20, it's possible XDR will ship A to B and cluster B gets set to n+10. XDR only guarantees data consistency if writes are 15 mins apart and your cross datacenter network doesn't go down.

The suggested method of solving this is to have two keys, one for each cluster, and XDR both keys. Then add them together in the app. You can maybe do it through a lua script, though I haven't tried.

Re: Call me maybe: Aerospike

#54
post #16

I currently work for a company that uses Aerospike quite heavily. In the past couple weeks, we have begun to notice data inconsistencies in our counters. We are seeing fluctuations in the data, despite having no decrement operations. We have the enterprise edition of Aerospike, allowing us to be in constant contact with their support team and developers. A couple weeks later, and we still have no idea why this is hap…

What are you migrating to ? As is away from Aerospike to... ? Thanks, very interesting anecdote/case.

We are first evaluating MongoDB. I believe the main reason behind this is we are already using Mongo in other parts of our application, so there is no additional setup when converting.

Note that nothing is set in stone. The decision to begin migrations only happened today. It is possible that we will end up using some other technology altogether, or even we find out the issues we are having with Aerospike and continue using that service.

Re: Call me maybe: Aerospike

#55
post #22

Earlier quoted context omitted.

Can you add any detail to this anecdote? It's interesting and important, but detail might help steer others appropriately. What kind of inconsistency? What kind of fluctuation?

I'm current working in AdTech. We are using counters to keep track of a lot of things that are important to us (e.g., reasons for not bidding on a given campaign, money spent within a given transaction, number of bid requests we get by exchange, etc). I personally have found two different, yet very similar, data fluctuations. The first I found when debugging an issue. I noticed the counters going up, dip down, then c…

Some of this stuff looks to me, as somebody not familiar with the domain of course, as a really good use case for event sourcing and in particular something Kafka/Samza could tackle well.

For future consideration and delayed evaluation of course. I guess if you absolutely must have the most up-to-date information so you can make decisions on it RIGHT NOW that wouldn't work very well :|

Or would it? If your bids and stuff are also going through the event stream..

Re: Call me maybe: Aerospike

#56
post #32
post #24

Earlier quoted context omitted.

Very strange. I've never encountered something like that with any database or cache. Wonder if it's somehow related to the way that your cluster is setup? How big is this cluster? Are you writing and reading to the entire cluster, or do you have certain nodes that you write to and others that you read from?

I have to admit, I'm not 100% on the entire configuration. However... We have two clusters of 8 nodes each. Each cluster is setup with 2 factor replication. The clusters are setup with cross datacenter replication. Your read / write question is a little hard to answer. In Aerospike, a given key will always reside on the same node, something to do with how they optimize their storage. Which means that anytime you writ…

Did you see this effect during split brain, or while all nodes were up and operating? I think you would have mentioned that what you saw was split brain.....

With datacenter replication, do you have a loop (multi-master) configured? Aerospike uses async replication, so using multi-master counters can be tricky....

Re: Call me maybe: Aerospike

#57
post #54

Earlier quoted context omitted.

What are you migrating to ? As is away from Aerospike to... ? Thanks, very interesting anecdote/case.

We are first evaluating MongoDB. I believe the main reason behind this is we are already using Mongo in other parts of our application, so there is no additional setup when converting. Note that nothing is set in stone. The decision to begin migrations only happened today. It is possible that we will end up using some other technology altogether, or even we find out the issues we are having with Aerospike and continu…

[deleted]

Re: Call me maybe: Aerospike

#58
post #49

Earlier quoted context omitted.

The impressions may be sparse, e.g. say you're retargeting CEOs (demographic information you're getting from a DSP) who have visited your website in the last month (via a pixel you drop) who are in New York City (via a geoIP DB). So, fine, a probabilistic model might work well. And you might decide to bid on 100% of impressions. And you might decide that you have to bid $200 CPM to win -- which you're OK doing, becau…

I'm really not seeing what you gain out of running fine-grained control all-the-time here. Even if it were vital for a customer that you hit a budget target exactly you could dynamically change the granularity of control as you got closer. If anything predictive modeling would give you better budget use when you do have the flexibility than granular adjustment would (I don't know much about the area though and just g…

A better example is frequency capping. Ever watch something on Hulu and see the same ad 4 times in a twenty-minute commercial? Or even, worse, back to back?

With a real-time data stack you can avoid the duplicated ad a good percent of the time. Better experience for buyers, for publishers, and for users.

Re: Call me maybe: Aerospike

#59
post #25

Is there any reason he's never tried to analyze a "classic" RDBMS like Oracle or SQL Server? I have to imagine they'd clobber a lot of this hipster technology.

Well, those systems are largely not distributed, barring SQL Parallel Data Warehouse and, very arguably, RAC.

Jepson tests network partitions... so less useful.

Re: Call me maybe: Aerospike

#60
post #54

Earlier quoted context omitted.

What are you migrating to ? As is away from Aerospike to... ? Thanks, very interesting anecdote/case.

We are first evaluating MongoDB. I believe the main reason behind this is we are already using Mongo in other parts of our application, so there is no additional setup when converting. Note that nothing is set in stone. The decision to begin migrations only happened today. It is possible that we will end up using some other technology altogether, or even we find out the issues we are having with Aerospike and continu…

You should take a look at "Call me maybe: MongoDB" [0]

[0] https://aphyr.com/posts/284-call-me-maybe-mongodb

Post reply on HN