81–90 of 110 posts
This is pretty much straight from the DynamoDB best practices. Offload infrequently accessed data (think time series data from previous months) to S3 and use another tool to query it. https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
Earlier quoted context omitted.
For a lot of teams, S3 is a data warehouse, and you can treat it just like HDFS for the most part with most things in the big-data ecosystem. Presto works well for letting you access it from these locations without having to explicitly import it (assuming it's in a traditional data warehouse or a common SQL DB).
I wonder if anyone here has a good heuristic for identifying the conditions under which using S3 + SQL layer as a data warehouse is a better choice than a SQL database? I've been exploring the former and it seems to only make sense if the size of your data is at a scale that is beyond what a single SQL database instance can handle, and even then, you can continue to scale out with systems like Citus so the limit isn'…
S3 + SQL is good for huge log/machine data, exploratory use cases that are not yet productionized, ELT (to get data from raw files into SQL, used as a feed to later layers), quick and dirty SQL against a directory of similarly structured files. I tend to think of it as a utility layer.
For long term analytics use, that involves a domain model, I’d still stick with dimensionally modeled (or snowflake) data warehouse techniques. Getting data into such a model can take weeks to months, so sometimes it might be better to do something quick and dirty in a data lake to prove a dataset or get a quick answer, vs. slow down the business waiting for a perfect model.
Lastly, I see storage + SQL as being the same conceptually as any RDBMS, with different performance, cost, and functionality. For example, SQL Server proprietary disk format + SQL Server query engine is somewhat analogous to Parquet + PrestoDB. In fact many proprietary vendors integrate with HDFS as a distributed storage layer for their proprietary formats which can be queried alongside open source storage formats by proprietary SQL query engines too.
Earlier quoted context omitted.
Feature-rich text search with elasticsearch is stupidly easy. It's a real shame.
I haven’t found this to be exactly the case - I recently worked with an engineer for more than a month tuning Elasticsearch (which required non-trivial changes to our catalog service). (It also took a good amount of effort to coordinate these changes across devices.) Like Goodreads, we have a fairly constrained catalog and we probably get a similar amount of queries. The search results are now much, much better, but…
I am about to start using ES for a project and just knowing which kinds of things could be useful to tune would be helpful.
Thanks!
how can I make my data more complicated? DynamoDB(non relational, key-value esque WTF-store) + something else
Isn't it expensive to use S3+Athena instead of DynamoDB?
However, I read a harrowing / awe-inspring blog post about someone doing just that. So...¯\_(ツ)_/¯
I use Goodreads as my main book tracker and reviewer. My impression is that Amazon bought it for whatever reason, and then ignored it. It seems like there's so much more potential to a social community around books than what Goodreads offers. The UI also never fails to disappoint. One of the core user features, searching for books, has all sorts of weird quirks. I'll use a very specific query that should guarantee a…
We agree. My co-founder and I built helloreads.com to address the UI concern above. We never felt like Goodreads got the necessary UI upgrade it needed. http://www.helloreads.com
IIRC you originally could add your own books to Goodreads which let you fill in the gaps.
[1] https://www.goodreads.com/book/show/1869210.La_insidiosa_fat...
[2] https://www.goodreads.com/book/show/101558.Harry_Potter_y_el...
I use Goodreads as my main book tracker and reviewer. My impression is that Amazon bought it for whatever reason, and then ignored it. It seems like there's so much more potential to a social community around books than what Goodreads offers. The UI also never fails to disappoint. One of the core user features, searching for books, has all sorts of weird quirks. I'll use a very specific query that should guarantee a…
Better integration with Audible (another Amazon property) seems like it would be a slam dunk. One of my largest frustrations with my audiobook habit is that there is no single source of truth for my to-read list. I need the books I'm interested in to be in an Audible wish list so I can get alerts when they're on sale, but then it's divorced from my currently-reading and completed indices in GR.
Earlier quoted context omitted.
Former Amazon employee here. They're cross-charged with lots of internal book-keeping. The rates teams "pay" internally are different from public pricing of course but expensive things externally are still expensive things internally. The cost-accounting used to be (3+ years ago) much more "just looking...no pressure to keep them down", but recently there's been huge efforts to bring internal AWS costs down, especial…
I'm reminded of a story my ex-Amazon friend would tell where it was a week-long process to get an extra $100-something stick of RAM for his workstation when he started there, needing various sign-off and escalations for such a grave expense. Meanwhile he was deploying some machine-learning categorization stuff he was working on to some cluster that would cost five figures of compute each time, and nobody batted an ey…
Also you have to jump through pretty extreme hoops if your hardware estimates (usually made at least 3 months out) were under-shot and now your service is redlining. This leads to teams way over-estimating their hardware needs and thus millions being spent on idle reserved EC2 capacity. So now they police (with savagery) idle capacity, so services or workloads that are "bursty" are basically an internal-bureaucracy nightmare.
It's much nicer to use AWS outside of Amazon :)
First lesson - it's taken Goodreads so long and so many users to actually start moving to microservices.
Earlier quoted context omitted.
The website search seems to not be tokenized, it seems to use the entire word only... so if there's a slight difference it won't find anything. The main thing I've noticed in Goodreads since they were bought is a strong focus on Kindle integration. The newer Kindles all have Goodreads integration, you can send notes etc. to the Goodreads profile, the site itself has changed very little (although Goodreads changed a l…
What drives me nuts is that the Goodreads integration never made it to Germany. It's frustrating to manually carry over the data while the code is just sitting there behind a country flag.