Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeWitt Clause, or can you benchmark %database% and get away with it

cube.dev

Post reply on HN
Print view View original

41–50 of 93 posts

Re: DeWitt Clause, or can you benchmark %database% and get away with it

#41

Post by Beltalowda »

danielheath wrote: post #38
Beltalowda wrote: post #30

Earlier quoted context omitted.

Looking at the language, almost all of them allow you to run benchmarks since it's phrased as "you may not publish benchmark results"; it doesn't forbid to actually run them. Never mind that MS-SQL, Oracle, etc. are not SaaS vendors of course. To be honest, if a cloud vendor has technical problems with someone running a few benchmarks then that would make me very wary of said cloud vendor. What's the difference betwe…

For a smaller/younger SAAS: If a customer environment is suddenly running at 100% of some resource when it wasn’t before, that’s an important thing to alert on / investigate. For established players it’s lost in the noise, but if it were me I’d appreciate a heads up for big changes.

Sure, a heads-up is certainly nice, but I don't think that running a (reasonable) set of benchmarks is all that out of the ordinary, or any different from just taxing the service at 100% with some periodic batch job or the like. Paying for it is even stranger IMO.

And for what it's worth, I did actually work for a few small SaaS businesses, but a few reasonable benchmarks wouldn't have been a problem.

Of course, if your benchmarks are going to take 50 hours it's a different story.

Also: I suspect a lot of these database SaaS services are a lot smaller than you'd might think. I know at least one of them is anyway because I worked there (and there's no DeWitt Clause).

Re: DeWitt Clause, or can you benchmark %database% and get away with it

#43

Post by floren »

You can add Splunk to the list of companies with a similar clause. As a Splunk competitor it makes sales a bit harder initially (we can show our product's numbers, but nothing to compare them against), but if you can convince customers to set up a head-to-head proof-of-concept of their own, well, they tend to figure out why Splunk doesn't want you publishing benchmarks...

Re: DeWitt Clause, or can you benchmark %database% and get away with it

#45

Post by HideousKojima »

l33t2328 wrote: post #33
HideousKojima wrote: post #21

Earlier quoted context omitted.

Not just lack of OSI approval, they're attempting to redefine the long accepted meaning of open source to include their new licenses. They want the goodwill of being "open source" without the obligations. The only sorts of licenses that have consistently been considered open source are either copyleft licenses like the GPL and do whatever the hell you want licenses like MIT and Apache. Do whatever you want... unless…

> unless you're part of a group the authors deem evil/immoral/unethical. What parts of the license mention that?

I wasn't referring to the BSL specifically with that comment, I was referring to recent attempts to call licenses like the Hippocratic License "open source" despite being anything but. The BSL is just a different example of a similar thing.

Re: DeWitt Clause, or can you benchmark %database% and get away with it

#46

Post by SahAssar »

Nextgrid wrote: post #7

If you want to benchmark for internal reasons you don't publish the results and nobody knows. If you want to make a service to the community, run your benchmarks, download Tor and publish the results anonymously. I don't see what the big deal is? Is this only limited to marketing claims where you post it on your company's website?

The only way I can openly talk about a service's performance is by doing it illegally and you don't find that weird?

Re: DeWitt Clause, or can you benchmark %database% and get away with it

#48

Post by jandrewrogers »

kstrauser wrote: post #27

Honestly, the presence of that clause screams to me "this app sucks and we'll sue you if you tell anyone how badly". That may not be the case whatsoever, but my first assumption is that they're trying to hide terrible performance.

While I would prefer if this clause was not a thing, I also understand why it exists even for great products.

It is surprisingly difficult to reproduce many workload benchmarks and quite easy to engineer a benchmark that misrepresents real-world database performance. There are tools that exist to generate optimally pathological workloads that target specific database implementations, while looking completely reasonable and innocuous. It doesn't even need to be a bad faith benchmark by a competitor, there is a high probability that the person configuring the environment does not know how to do it correctly and/or optimally.

The DeWitt Clause is a defense against the unfortunate pervasiveness of incompetent and/or bad faith benchmarking. Companies have a well-founded reason to not trust third parties to do a good job of representing the performance of their product.

Re: DeWitt Clause, or can you benchmark %database% and get away with it

#49

Post by somat »

josephcsible wrote: post #17
chmod775 wrote: post #8

It would be quite refreshing if we could have a story in which Oracle are the good guys for once. I'm sure they are at least purchasing some modern-day 'indulgences' by - for instance - donating food to starving north korean elites?

If Oracle ever wants to be the good guys just once, I have an idea for them that's right in their wheelhouse. Step 1: buy grsecurity's kernel hardening patches. Step 2: put said patches in the publicly released UEK source. Step 3: wait for grsecurity to refuse to give them future patches. Step 4: sue grsecurity for imposing further restrictions on the exercise of rights granted by the GPL.

I think the weakness of your model is just because you have the right to distribute a certain patch level that does not mean you automatically have the right to distribute further patches,

Conversely, If the right to distribute is revoked, say a GPL to closed source license change you have still the right to distribute any versions originally distributed under the open license.

A good example off all this is the sordid history of the berkely db.

Re: DeWitt Clause, or can you benchmark %database% and get away with it

#50

Post by kstrauser »

jandrewrogers wrote: post #48
kstrauser wrote: post #27

Honestly, the presence of that clause screams to me "this app sucks and we'll sue you if you tell anyone how badly". That may not be the case whatsoever, but my first assumption is that they're trying to hide terrible performance.

While I would prefer if this clause was not a thing, I also understand why it exists even for great products. It is surprisingly difficult to reproduce many workload benchmarks and quite easy to engineer a benchmark that misrepresents real-world database performance. There are tools that exist to generate optimally pathological workloads that target specific database implementations, while looking completely reasonab…

I understand your point, and that’s a reasonable argument. I do disagree with it, though.

Imagine a hypothetical FooDB by Bar, Inc. If Bar never put that clause in the FooDB license, then I think you’re absolutely right. People would come up with some awful-looking benchmarks that made it look bad. However, what a golden opportunity for Bar! They could step up with some free or steeply discounted consulting to help the benchmarker fix the problem and publish new, good results. They wouldn’t have to do that too many times for word to get around on sites like this: FooDB is nice and fast when you tune it correctly! That would come along with some enormous goodwill, and also the assumption that if your FooDB installation is performing poorly, then it must be your fault because all the benchmarks say it’s really fast for everyone else.

I’m not going to tell Bar what their business model should be. I have my thoughts on it, but it’s their business to run as they see fit. But if I see Bar being open and helpful with a freely-accessible tech blog telling you how to make FooDB stand up and dance, I’ll tend to believe that it’s probably an interesting product to look at. If they guard those secrets behind a wall of lawyers and sue people who speak ill of FooDB, I’ll tend to believe they’ve got something to hide. Either one of those beliefs might be completely wrong, but that’s still how I’m likely to perceive it.

Post reply on HN