Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mito – Excel-like interface for Pandas dataframes in Jupyter notebook

trymito.io

Post reply on HN
Print view View original

91–100 of 101 posts

Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook

#91

Post by kite_and_code »

NoImmatureAdHom wrote: post #79
kite_and_code wrote: post #63

Earlier quoted context omitted.

bamboolib co-founder here: It's correct that bamboolib is (still) closed-source (which might be subject to change but I don't make promises). It's also correct that customers can extend the bamboolib UI in various ways via plugins that they can author themselves. That empowers them to build bamboolib into the kind of tool that they want. Also, all the code is always exported and thus, there is at least no "code locki…

I'm sure you have good intentions, but the fact of the matter is the company may be acquired or the people replaced, and those intentions might change. IMHO investing in a closed-source product like bamboolib as a tool for an important business function is very risky. Imagine you're a small company, and you start using bamboolib for some part of your data analysis pipeline. Bamboolib gets acquired (you have exited ki…

Fair points.

I guess in this specific case at hand, companies can switch between bamboolib, mito, dtale and it is less likely that all of them will become unavailable at the same time. The switch is also not so hard because there are no underlying proprietary file formats involved (except for bamboolib plugins) because the generated code is pandas, plotly, etc.

Similarly as described below/above: counter-intuitively, the availability of open-source LibreCalc makes it easier and safer to adopt closed-source Excel.

Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook

#92

Post by kite_and_code »

narush wrote: post #76
kite_and_code wrote: post #52

Earlier quoted context omitted.

I love how mito enables companies to use the power of open-source! You might want to think about enabling companies to create the company specific extensions themselves e.g. via a plugin API. You might still request them to pay for this version of Mito but they are enabled to extend it with their engineering power instead of relying on you. We had good experiences with this at bamboolib (I am one of the co-founders)…

Yeah, we've thought a bit about a plugin API - for the reasons you say, I think it would be an awesome feature to open up to teams! Any tips on going about it? No need to share the secret sauce, unless you want :P To be totally honest, we're not architected super well to support plugins currently. The big challenge would be allowing users to specify this plugin in pure Python (seems like we want this) - but we think…

Cool to hear that!

To be honest, we regularly refactor our architecture at bamboolib in order to make sure that there is almost no gap between what we would love to say in natural language and the code that we need to write.

This resulted in a very stable and clear internal API surface (read architecture). So, literally, all we had to do was adding mount points where users could register their plugins and then include those at render time.

The next day, customers could write plugins just as we did. And, as a matter of fact, all the bamboolib transformations, visualizations, views, etc are just sophisticated plugins that our customers could write themselves because they have access to the same API as we do.

So, no secret sauce except for "good architecture" which is easiest achieved as an ongoing effort rather than an one-off project.

Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook

#93

Post by kite_and_code »

Closi wrote: post #82
kite_and_code wrote: post #50

Earlier quoted context omitted.

Yes, I am one of the co-founders of bamboolib and employed by Databricks. I already added my disclosure to the following answer [0] in this thread but I was hesitant to add it to every answer. Do you prefer if I explicitly add my affiliation in every comment that mentions bamboolib? If so, I will try to edit them (if the HN UI still allows me to - I observed that it stops allowing this after some time) [0] https://ne…

> Do you prefer if I explicitly add my affiliation in every comment that mentions bamboolib? Personally I thought your original post's tone implied that you weren't affiliated to me personally. You don't have to add a formal 'disclosure', but you could just say "I built x which is..." rather than "Another product is x which is...".

Thank you for sharing that observation and the suggestion! I will keep that in mind :)

Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook

#94

Post by kurupt213 »

NoImmatureAdHom wrote: post #79
kite_and_code wrote: post #63

Earlier quoted context omitted.

bamboolib co-founder here: It's correct that bamboolib is (still) closed-source (which might be subject to change but I don't make promises). It's also correct that customers can extend the bamboolib UI in various ways via plugins that they can author themselves. That empowers them to build bamboolib into the kind of tool that they want. Also, all the code is always exported and thus, there is at least no "code locki…

I'm sure you have good intentions, but the fact of the matter is the company may be acquired or the people replaced, and those intentions might change. IMHO investing in a closed-source product like bamboolib as a tool for an important business function is very risky. Imagine you're a small company, and you start using bamboolib for some part of your data analysis pipeline. Bamboolib gets acquired (you have exited ki…

This is a flawed way of looking at things.

Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook

#95

Post by kurupt213 »

santiagobasulto wrote: post #12

I like this. Is a "friendlier" way to browse data. Said that, I have to add: Exploring large datasets requires a COMPLETELY different mindset. When your data starts growing, it's impossible to keep it all in a visual format (for 2 reasons[0]) and you have to start thinking analytically. You have to start looking at the statistical values of your data to understand what's its shape. That's why the `.describe()` and `.…

In all seriousness, excell can’t be the right option for 15GB of alphanumeric data (one sheet?)

Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook

#96

Post by kurupt213 »

awild wrote: post #23
santiagobasulto wrote: post #12

I like this. Is a "friendlier" way to browse data. Said that, I have to add: Exploring large datasets requires a COMPLETELY different mindset. When your data starts growing, it's impossible to keep it all in a visual format (for 2 reasons[0]) and you have to start thinking analytically. You have to start looking at the statistical values of your data to understand what's its shape. That's why the `.describe()` and `.…

> try loading a 15GB CSV in Excel. Or visualising it in r or pandas without meaningful subsampling.

I have an excel template for handling a relatively large amount of data. No where 15GB on one sheet. I use it for preprocessing experimental data from a single experiment. There are about 10 chart tabs build in so I can visually inspect the data looking for errors (and go back and inspect the raw instrument data when something looks off).

The aggregate data is around 1.5 million experimental results. MiniTab is too unwieldy and requires too much manual reformatting of the data sheets.

Is this something I should be looking at in R or project Jupyter? Does one make better visualizations than the other?

Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook

#97

Post by awild »

kurupt213 wrote: post #96
awild wrote: post #23

Earlier quoted context omitted.

> try loading a 15GB CSV in Excel. Or visualising it in r or pandas without meaningful subsampling.

I have an excel template for handling a relatively large amount of data. No where 15GB on one sheet. I use it for preprocessing experimental data from a single experiment. There are about 10 chart tabs build in so I can visually inspect the data looking for errors (and go back and inspect the raw instrument data when something looks off). The aggregate data is around 1.5 million experimental results. MiniTab is too u…

Ggplot is extremely powerful if you can grok its grammar, which takes some getting used to. But I'd assume that if you see a graph in a scientific paper it's made with ggplot.

Having many data points you want to explore you are always going to be at the edges of what your hardware and software can produce.

The last really big datasets I worked with were for my thesis and I had to do subsampling to below 10% to get results within 10minutes or so and that was basically plotting midi recordings of piano performances, so nothing gigantic

Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook

#99

Post by MaxDPS »

harabat wrote: post #18

For those who are going through the thread finding new tools: pandas-profiling[0] is a library for automatic EDA (which bamboolib[1], mentioned elsewhere, also does). [0]: https://github.com/pandas-profiling/pandas-profiling [1]: https://bamboolib.com/

I just found out about pandas-profiling a couple days ago and the examples blew my mind, it looks amazing (I’ve yet to actually try it out though).

Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook

#100

Post by narush »

imcoconut wrote: post #90
narush wrote: post #83

Earlier quoted context omitted.

Mito is open source, but using Pro features does actually require a Pro or enterprise license. You can check out this callout in the license [1], as well as the restrictions on Mito Pro features here [2]. We're in the process of fixing up the upgrade to Pro process a bit... as you can tell... :) You can of course fork Mito and turn off telemetry as long as you open source your changes! Go for it - happy to hop on a c…

Mito looks awesome. Just want to say that I respect the fact that you built this library, are offering it open source, for free, and want the telemetry on. You are up front and open about it. Sometimes I think people can get a little entitled with all the work someone else puts in to a project they want to use (not accusing GP of this, speaking generally). As you said, under the license anyone is more than welcome to…

Thanks a ton!
Post reply on HN