181–190 of 199 posts
Earlier quoted context omitted.
This "single box" has a lot of computing units as on AWS EC2.
The figure I saw around maybe 2015 was that AWS (maybe Heroku) is 10x more expensive and bare metal is 10x more efficient/faster, lending a 100x difference in price-performance. Both factors are probably larger nowadays.
I'm quite tired of everyone wanting to build "large scale systems" and play at being Netflix. The truth of the matter is the vast vast majority of people will never need to do this with their project and instead will just end up making an expensive to maintain mess with way too many moving parts. At least as important as designing something that can scale up is designing something that can scale down . You don't know…
I'm quite tired of everyone wanting to build "large scale systems" and play at being Netflix. The truth of the matter is the vast vast majority of people will never need to do this with their project and instead will just end up making an expensive to maintain mess with way too many moving parts. At least as important as designing something that can scale up is designing something that can scale down . You don't know…
So true. A transparently priced PaaS service would have fixed this problem long back. Most of the users can then simply trust the service to automagically scale up if at all when required and pay on the same lines of a custom configured IaaS solution.
Earlier quoted context omitted.
I didn't get any real interest in my office, but I applied TLA+ to a debugging challenge. I described a well-specified system (abbreviated, for instance data was just "DATA", not the actual myriad number of message types that could be sent). The intention was to understand a bug and possibly identify which of three suspected locations it was occurring in. I had enough of the specification for each part implemented: T…
That's awesome! I'm adding this write up to my collection of case studies. You should consider doing a blog post with a mockup problem really similar to illustrate that process with code and specs.
I'm quite tired of everyone wanting to build "large scale systems" and play at being Netflix. The truth of the matter is the vast vast majority of people will never need to do this with their project and instead will just end up making an expensive to maintain mess with way too many moving parts. At least as important as designing something that can scale up is designing something that can scale down . You don't know…
Oh interesting, I have never seen Anki ( https://apps.ankiweb.net/ ) being used for large blocks of source code. Anki is an open source application (desktop + mobile) for spaced repetition learning (aka flashcards). It's a very popular tool among people who want to learn languages (and basically anything else you want to remember). There are many shared decks ( https://ankiweb.net/shared/decks/ ). Creating and format…
I've been experimenting with Anki recently. I loaded up a deck of popular fonts, with the goal being to memorize them to the point where I could recognize them in the real world. Each card contains the sentence, "The quick brown fox jumped over the lazy dog," and you have to identify what font it's written in. The first day was really tough; I missed cards over and over again. 20 new cards (the default) is probably t…
There are a lot of fonts out there (~50,000 families according to random Quora people). Their distribution is probably power-law-like even if you discount the ones that are preinstalled on major platforms. It might make sense to recognize a few if you want to be able to really deeply discuss the difference in how they are used for design, but just recognizing them doesn't seem like the right way to gain that understanding. If you repeatedly perform a task where you have to recognize a font, learning only the top 100 won't help you much since it will eventually become pretty obvious. If you don't do that task, then why train for it instead of looking it up as necessary?
My thinking on "what's worth flashcarding" is that there are two major categories where it makes sense. First, if you need to remember a bunch of specific facts and you will need to recall them more quickly than they can be looked up. This is the case for things like tests, but there are also reasonable possibilities for this in real work (for example, if you are a programmer you may know you are going to need to look up the parameter ordering of a standard library method that you use only once a month, or you could memorize it).
The second is where you are using the flashcards as a scaffold, but the actual knowledge is something that references or brings together the facts that are contained in the flashcards. Recognizing fonts fits into this category, but I have a hard time imagining that actually recognizing them is the knowledge that is most efficient. Instead maybe you should be studying the major categories of fonts, features of fonts, or something that would help you make quicker decisions for whatever the real task is. I used to be able to recognize a lot of fonts and it's basically only useful as a parlor trick.
Although if you are new to design then learning the top 20 or whatever could be helpful to just have a basic fluency with Arial vs Times New Roman vs Comic Sans, so you have a shared vocabulary to discuss with others. "It's like Times New Roman but more suited for headlines and all caps" for example.
This is the weird confluence of work I've done at multiple companies (in one case I basically implemented a SRS like Anki with applications to finance exams, and in another I did a lot of work with fonts for a laser cutting design editor).
I'm quite tired of everyone wanting to build "large scale systems" and play at being Netflix. The truth of the matter is the vast vast majority of people will never need to do this with their project and instead will just end up making an expensive to maintain mess with way too many moving parts. At least as important as designing something that can scale up is designing something that can scale down . You don't know…
In general, the 'get it done' mentality is the one that makes economic sense, because once you've added together the pile of software that doesn't need to scale to the other piles where this long-view doesn't matter, you have almost everything built. The other piles, for the record, include software:
- that is designed wrong so it needs to be re-written
- is obsoleted by changes in business direction (a project canceled, for example)
- gets replaced by something off-the-shelf or open-source
- is built for a startup that won't survive, or that gets aqui-hired, or that pivots to a wildly different thing
On the other hand, I sometimes see the opposite thing in heavily analytical work, where data science work is done in Python because its "easy", and then a team of engineers builds a crazily complex pipeline to make the python perform in some reasonable time frame. (Hi, Spark!). In my workplace, one example allocates bits of a job to roughly 100 machines, moving data to each, in a cloud environment where the data movement overhead is constantly fighting the benefits of distribution.
Earlier quoted context omitted.
> Few companies will take a product that actually needs large scale systems and hire someone that has no prior experience. No I think most people end up hiring those who have experience creating big complicated systems but haven't stuck around long enough for their chickens to come home to roost.
That's an important point, especially with the oft-repeated statistic of 2-years as the average tenure of an engineer. Of course, averages (even if true) are like stereotypes. It would be interesting to see the tenure data on the experts (consultants/implementers) of large-scale systems, other than at the iconic ones (e.g. Google, Netflix).
Earlier quoted context omitted.
I think you highly overestimate it. Most large companies are a series of small groups that act as companies that have nearly trivial (to the point of absurdity) engineering concerns. Sure, a few groups in each F500 need epic skills - but I think that's an exceedingly vanishing amount of the (actual) work that is being done. The term Enterprise and what it stands for earned it's laughable reputation for a reason.
I've worked at smaller companies. Even still, it's not hard to hit relatively large amounts of data, depending on the field. As an example, any kind of analytics could generate terabytes of data a day... per customer. A side project I am building will have to handle billions of events per day. Per customer. There are 0 customers (this is for fun, not profit), but as soon as it would hit one customer I would need to c…
Wait, what? I've never worked anywhere where one customer generated terabytes of data per day, and I've worked on very large commercial enterprise software.
The only thing I have experience with that produces anything close to that kind of data per customer is in genetic sequencing, and you only do a customer once. (Even that isn't a TB in its bulkiest, raw data form, and the formats used for cross-customer analysis are orders of magnitude smaller).
> For many of us, I imagine we've spent a lot of time fighting fires at organizations where one service going down was a serious problem, causing other services to fail, and setting your infrastructure ablaze.
The reason so many of us have worked in places like that is that those places 'got stuff done' and survived and grew.
I'm quite tired of everyone wanting to build "large scale systems" and play at being Netflix. The truth of the matter is the vast vast majority of people will never need to do this with their project and instead will just end up making an expensive to maintain mess with way too many moving parts. At least as important as designing something that can scale up is designing something that can scale down . You don't know…