161–170 of 376 posts
Of course, it won't work out to replace 1000 Elasticsearch servers - that's where the advantage of a true "big data" tool will show - but none of the clients really have data "that big".
This is precisely the point made by McSherry, Isard and Murray in their lovely paper, "Scalability! But at what COST?" (Usenix HotOS '15). They demonstrate how much performance headroom there is in modern CPU and memory, and show how simple cache-sensitive batch algorithms running on a single core can outperform hundreds of cores running distributed map-reduce style jobs. https://www.usenix.org/system/files/conferenc…
But nobody in BigCo(tm) would do that, because everyone (including the executives) involved want to have Hadoop/BigData(tm) in their resume. /sarcasm
Earlier quoted context omitted.
SO content changes all the time. Votes, comments, moderation, edits, tagging, search and recommendations, etc. There are also real-time community features. It's not as simple as it seems.
They said read heavy, not read only. I guarantee you the bulk of traffic to SO is hitting a page and performing zero writes.
Yes it's read-heavy but there's still plenty of work done in assembling a page. It's definitely not as simple as caching at the CDN edge for every hit.
Earlier quoted context omitted.
If you're parsing market data then you shouldn't really be allocating at all once the initial setup is complete. So there shouldn't be a need for ref counts. This is because the allocator itself can have an unbounded runtime that takes milliseconds, causing you to drop. In the past I've replaced malloc with an implementation that asserts if called on certain threads after init time.
Milliseconds ??! What causes that to occur? Though I suppose if they can take that long, avoiding them is an even better idea than it is normally... For constant time allocations, try TLSF: http://www.gii.upv.es/tlsf/
It's one benefit of a gced language with compaction, allocations are typically bounded (except when they trigger a gc).
Earlier quoted context omitted.
> That means we transfer 55 TB data / month this isn't a lot. Helps that site is mostly text data. I know of a relatively small cloud security system that transfers petabytes/month to/from a handful of customers. 4 ingest pods, 8 pipeline pods, 7 time-series db servers, 2 sql servers
The amount of data transferred is meaningless to compare. SO is a relatively complex site with dynamic content and real-time features. Compared to other similar sites like Reddit or Quora which are far slower yet running on more hardware, it shows what proper efficient architecture and code can do.
Earlier quoted context omitted.
If you are writing C++, any field! Just use std::shared_ptr and std::unique_ptr from the standard library, along with std::make_shared and std::make_unique.
std::shared_ptr uses CAS atomics, which in heavy multithreaded code (multiple threads operating on the same pointers), can have surprising overhead in some situations.
Earlier quoted context omitted.
If you are writing C++, any field! Just use std::shared_ptr and std::unique_ptr from the standard library, along with std::make_shared and std::make_unique.
This is the quickest way to kill your performance in C++. I guarantee it wasn’t what Carmack was talking about. I used sharedptr extensively in a game engine. Whoops: suddenly 20% of the frame time was gone, never to be recovered. Once that performance is gone it’s almost impossible to get it back, short of rewriting every system.
Earlier quoted context omitted.
The amount of data transferred is meaningless to compare. SO is a relatively complex site with dynamic content and real-time features. Compared to other similar sites like Reddit or Quora which are far slower yet running on more hardware, it shows what proper efficient architecture and code can do.
Reddit threads are also much hotter for writing and have O(thousands) of replies.