Is there an alternative to BigTable which written in C or Go ?
ScyllaDB, which is basically a C++ Cassandra reimplementation, which in turn borrows a lot of design ideas from BT.
Xg2xg: Lookup table of similar tech and services for ex-Googlers
111–120 of 133 posts
Re: Xg2xg: Lookup table of similar tech and services for ex-Googlers
#112Earlier quoted context omitted.
I keep seeing that Borgmon is deprecated but no references to what the replacement is like. Can someone please share even a high level summary? Is the replacement a high cardinality event store like Facebook’s Scuba [0]? [0] https://research.fb.com/wp-content/uploads/2016/11/scuba-div...
The replacement (Monarch) is similar to borgmon except: * All metrics have an associated type. Eg. Response time (milliseconds). That's great because units for derived metrics can be dynamically computed. Eg. Bytes/second. * The query language can fairly efficiently compute metrics at query time rather than needing everything precomputed (eg. 95 percent latency across 1000 tasks can be calculated in real-time). * The…
Importantly, Monarch is push-based and centralized. Previously, product teams would have to run their own borgmen, and those in turn would get scraped by the upstream borgmen of their orga for aggregation, archiving etc. Monarch is more of an As A Service offering.
Re: Xg2xg: Lookup table of similar tech and services for ex-Googlers
#113I had never thought about this before but from this repo I can see all these smart engineers at the big tech firms live within their own walled garden (or prison?). If it weren’t for such a vibrant open source world then they’d really be stuck if they tried to apply their knowledge outside of their companies. Thanks for sharing the repo!
They don’t, and if you’ve got the G-factor, pun intended, to get an offer at Google you can get an offer anywhere. Which, honestly, makes this repository all the more confusing to me. I’m sure there are some people that have lived only within this ecosystem, but that’s the possibly the height of privilege.
I've had companies reject me explicitly because they had bad experience with other googlers. I'm honestly considering dropping it from my CV.
Re: Xg2xg: Lookup table of similar tech and services for ex-Googlers
#114Earlier quoted context omitted.
Not sure why you think free food is traded for free time. Free food helps me use my time at work more effectively. I usually have breakfast and lunch with teammates, and it's a mix of socialization and work-related conversations. Half an hour in the free cafe > 45-90 minutes going someplace to buy food.
So do you get to go home 45-90 minutes earlier then? I think that is what the original comment was questioning.
I typically work ~8-4, with some occasional time in the evenings. So, yes.
Re: Xg2xg: Lookup table of similar tech and services for ex-Googlers
#115Earlier quoted context omitted.
> The query language can fairly efficiently compute metrics at query time rather than needing everything precomputed I'm having hard time figuring out how query language might affect exectuion time. Granted, I've never seen borgmon in action, so the first question should probably be: how different its query language is from what Prometheus offers? If it's more on imperative side than declarative one then your point m…
I think what he is saying is that it is easier to do something ad hoc in Monarch since it is a global service with unified storage and retrieval. A Googler can join the data of Gmail and Ads easily if they want, whereas combining the borgmon data of those services is either not possible or it involves downloading CSVs and performing the join externally. > Does it allow different aggregation methods (avg/sum/max/whate…
No, I think it's strictly about where aggregations and other computations occur. You hint at that when you mention CSV files. Borgmon keeps X hours of data in memory and looks up anything older from TSDB. I never understood how to aggregate data from the same service, after the fact, in a reliable fashion. Or if it was possible at all. I believe it was only working when looking at data in memory. That's why people had rules to aggregate data at multiple levels as it was being ingested, then persisting that into TSDB.
I don't remember the latter being much more sophisticated than a bunch of files in GFS with the timeseries and some metadata, either. Monarch, I think, is based on Bigtable. It has a richer API. It might even use BT coprocessors to perform computations closer to the where the raw data is stored, rather than in the Monarch frontends. (I haven't watched the public talk, so I'm just going from vague recollection.)
Re: Xg2xg: Lookup table of similar tech and services for ex-Googlers
#116Earlier quoted context omitted.
The fact that Monarch configs can be written in Python instead of Borgmon is a huge win for our team; being able to write and debug our own alerting rather than have to bug SREs every time has been worth the switch alone.
Monarch, like Borg, is configured by RPC. You can use python if that suits you but you can also use C++ or Java or Go or any language capable of putting an encoded protobuf on the wire. This is a mistake people also make about Borg: there is borgcfg but borgcfg is not Borg's API. You can use the Borg RPC interface from any language and borgcfg is optional. Compare to borgmon where use of the DSL is obligatory.
We're specifically using Gmon (GMon/Viceroy for dashboards, GMon Monarch for alerting), which is all Python built on top of Monarch.
Re: Xg2xg: Lookup table of similar tech and services for ex-Googlers
#117Earlier quoted context omitted.
I think what he is saying is that it is easier to do something ad hoc in Monarch since it is a global service with unified storage and retrieval. A Googler can join the data of Gmail and Ads easily if they want, whereas combining the borgmon data of those services is either not possible or it involves downloading CSVs and performing the join externally. > Does it allow different aggregation methods (avg/sum/max/whate…
No, I think it's strictly about where aggregations and other computations occur. You hint at that when you mention CSV files. Borgmon keeps X hours of data in memory and looks up anything older from TSDB. I never understood how to aggregate data from the same service, after the fact, in a reliable fashion. Or if it was possible at all. I believe it was only working when looking at data in memory. That's why people ha…
Borgmon/TSDB was also a bigtable, an enormous one known as 'dumptruck'.
Re: Xg2xg: Lookup table of similar tech and services for ex-Googlers
#118Earlier quoted context omitted.
Prometheus is borgmon. Which is a bit funny as it's used by k8s now. It's also funny as the author left Google, wrote it in 2 years for SoundCloud, then went back to Google. https://prometheus.io
Why is the part that k8s uses it funny? That would have been one of the goals right? K8s is like lot of these projects except that it was created as a open source version of borg from within Google.
I feel like much of the following conversation violates NDAs signed as a condition of employment there. Is this not the case?
Re: Xg2xg: Lookup table of similar tech and services for ex-Googlers
#119There is no listed equivalent of RecordIO. What do people use for high-reliability journals? When I needed something like RecordIO to store market data, I couldn't find anything. So I implemented https://github.com/romkatv/ChunkIO . I later learned of https://github.com/google/riegeli (work in progress), which could've saved me a lot of time if only I found it earlier. I think my ChunkIO is a better though.
That appears to be exactly RecordIO. It even has transposition. Is there a reason that doesn't meet your requirements? Edit: it even includes an open source implementation of Cord, which they've renamed to Chain for some reason.
> That appears to be exactly RecordIO.
I suppose you mean "exactly" in a figurative way. Riegeli is definitely inspired by RecordIO and is meant as a successor to it but it's not RecordIO.
> Is there a reason that doesn't meet your requirements?
I need to store timeseries with fast lookup by timestamp. Riegeli doesn't support this out of the box. If I had discovered it before I built ChunkIO, I probably would've pulled the low-level code out of it and added timeseries support on top. Or maybe not. Reliability is very important to me and it's risky to use work-in-progress software that may or may not have any production footprint (I'm no longer with Google so I don't know if they use it internally.)
Re: Xg2xg: Lookup table of similar tech and services for ex-Googlers
#120Earlier quoted context omitted.
That appears to be exactly RecordIO. It even has transposition. Is there a reason that doesn't meet your requirements? Edit: it even includes an open source implementation of Cord, which they've renamed to Chain for some reason.
> That appears to be exactly RecordIO. I suppose you mean "exactly" in a figurative way. Riegeli is definitely inspired by RecordIO and is meant as a successor to it but it's not RecordIO. > Is there a reason that doesn't meet your requirements? I need to store timeseries with fast lookup by timestamp. Riegeli doesn't support this out of the box. If I had discovered it before I built ChunkIO, I probably would've pull…
I don't understand. RecordIO doesn't support lookup of any kind; it is a linear format. The interface of Riegeli looks to me exactly like the interface to RecordIO. All they've done is removed support for Google's abstract File* storage interface so it can be used by the public.
What you are describing sounds like SSTable. Perhaps you could benefit from LevelDB.
https://www.igvita.com/2012/02/06/sstable-and-log-structured...