Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
1–10 of 38 posts
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#21. This looks super cool
2. I admit to not understanding data lakes at all. I thought it was like a failure case for like, "we can't figure out how to get this data into a database", because isn't updating it a huge chore? You have to make sure that if you're updating you're not also generating new analytics, which it seems like you're always doing because it's very slow. Don't databases solve this pretty elegantly? Why are there all these tools for dealing with data in flat files?
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#31. This looks super cool 2. I admit to not understanding data lakes at all. I thought it was like a failure case for like, "we can't figure out how to get this data into a database", because isn't updating it a huge chore? You have to make sure that if you're updating you're not also generating new analytics, which it seems like you're always doing because it's very slow. Don't databases solve this pretty elegantly?…
2. I hate the name data lake but I’ve always used them as part of a pipeline. It’s useful to keep the raw data around when you need to recover / replay the pipeline.
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#41. This looks super cool 2. I admit to not understanding data lakes at all. I thought it was like a failure case for like, "we can't figure out how to get this data into a database", because isn't updating it a huge chore? You have to make sure that if you're updating you're not also generating new analytics, which it seems like you're always doing because it's very slow. Don't databases solve this pretty elegantly?…
A database is typically an alive, running program that requires maintenance. There is a strong coupling for most “databases” between disk format and executable code. It’s not easy to read from a random Postgres database sitting on disk. You could not do in Python, “import postgres” and then “postgres.open(‘mydb’)”.
I’m no data scientist, and have only worked with data lakes a couple times, but I can see why data science tends to be done with very predictable (if inefficient) data formats such as CSV, JSON, and JSONL.
Edit: SQLite is the best of both worlds. It’s a database, but it’s also “just a file.” It’s easy to work with, and many languages & frameworks are getting good support for it. SQLite’s reliability-first approach means many of the kinks that arise from involving databases (so much complexity!!) are ironed out and don’t arise as issues. (Things like auto-indexing, auto-vacuuming, avoiding & dealing with corruption, backwards & forwards compatibility, …)
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#51. This looks super cool 2. I admit to not understanding data lakes at all. I thought it was like a failure case for like, "we can't figure out how to get this data into a database", because isn't updating it a huge chore? You have to make sure that if you're updating you're not also generating new analytics, which it seems like you're always doing because it's very slow. Don't databases solve this pretty elegantly?…
One side of data lakes is it's more of a start of where the data is. Your normalised and more processed data may end up in a nice clean database but the start is not like that.
The other main points are usually
* Data size
* Data access patterns
* Data formats
The more you're looking at "I want to pull 400G of data out of my 30TB set of images from a bunch of machines running a custom python script, then shut it down in twenty minutes and not start anything else until tomorrow" then the more a data lake makes sense vs a database.
> because isn't updating it a huge chore?
Not with the right tools, which can also give you things like a git-like commit experience with branching.
> You have to make sure that if you're updating you're not also generating new analytics, which it seems like you're always doing because it's very slow.
Why would you be generating new analytics? I feel I've missed something there.
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#6Finally a decent name.
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#71. This looks super cool 2. I admit to not understanding data lakes at all. I thought it was like a failure case for like, "we can't figure out how to get this data into a database", because isn't updating it a huge chore? You have to make sure that if you're updating you're not also generating new analytics, which it seems like you're always doing because it's very slow. Don't databases solve this pretty elegantly?…
2. I hate the name data lake but I’ve always used them as part of a pipeline. It’s useful to keep the raw data around when you need to recover / replay the pipeline.
I think most data lakes really turn into a data swamp as people/orgs are not diligent enough to keep them in a good shape. In absence of costs, people never delete anything voluntarily and the garbage will grow monotonically until it takes up all space.
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#8Earlier quoted context omitted.
2. I hate the name data lake but I’ve always used them as part of a pipeline. It’s useful to keep the raw data around when you need to recover / replay the pipeline.
I think most data lakes really turn into a data swamp as people/orgs are not diligent enough to keep them in a good shape. In absence of costs, people never delete anything voluntarily and the garbage will grow monotonically until it takes up all space.
That applies to databases as much as data lakes.
I don't know why it's so hard to get across that data expires in the same way that language changes.
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#9Finally a decent name.
Hello! I am one of the maintainers of Daft. Funny enough I just gave a presentation about Daft in London and we all had quite a laugh at the name :D
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#10So Daft is a distributed Polars ?
If we set apart the distributed part, what's the "killer feature" of Daft for trying to compete with Polars (and Pandas) ? Are they API-compatible ? How's the memory consumption benchmark ? (TBH, this is the only interesting metric. Timing and Latency are not really important when your most important competitor is Spark)