The telemetry thing is... weird. So we can use it for free but have no way to turn it off but upgrade to paid?
Yeah, I have a hard pass on anything that offers an "Open Source" version, but actually meant to be a "Try it and be my Guinea Pig".
Mito – Excel-like interface for Pandas dataframes in Jupyter notebook
11–20 of 101 posts
Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook
#12I like this. Is a "friendlier" way to browse data. Said that, I have to add:
Exploring large datasets requires a COMPLETELY different mindset. When your data starts growing, it's impossible to keep it all in a visual format (for 2 reasons[0]) and you have to start thinking analytically. You have to start looking at the statistical values of your data to understand what's its shape. That's why the `.describe()` and `.info()` methods in Pandas are so useful. After many years doing this, I can "see" the shape of my data just by looking at the statistical information about it (mean, median, std, min, max, etc).
After some time you don't need to rely on visual tools, just can run a few methods, look at some numbers, and understand all your data. Kinda feels like the operator of The Matrix that is looking at the green numbers descend and knows what's going on behind the scenes.
[0] Your eyes are really inefficient at capturing information and there's only so much memory available: try loading a 15GB CSV in Excel.
Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook
#13If others are interested, Mito does not work in vscode or Google Collab. Only classic Jupyter Notebooks and Jupyter Labs are supported currently [1]. [1] https://docs.trymito.io/misc/faq
Thanks for checking that! I use vscode so this is a no go...
Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook
#14I like this. Is a "friendlier" way to browse data. Said that, I have to add: Exploring large datasets requires a COMPLETELY different mindset. When your data starts growing, it's impossible to keep it all in a visual format (for 2 reasons[0]) and you have to start thinking analytically. You have to start looking at the statistical values of your data to understand what's its shape. That's why the `.describe()` and `.…
Do you as a rule look at a sample of the individual raw data, non aggregated?
Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook
#15I like this. Is a "friendlier" way to browse data. Said that, I have to add: Exploring large datasets requires a COMPLETELY different mindset. When your data starts growing, it's impossible to keep it all in a visual format (for 2 reasons[0]) and you have to start thinking analytically. You have to start looking at the statistical values of your data to understand what's its shape. That's why the `.describe()` and `.…
Do you as a rule look at a sample of the individual raw data, non aggregated?
Usually aggregated... then can start looking at "subsets". For example, step 1 is look at the whole dataset. Then you identify that there are a lot of rows with a type of missing value, so you look at the statistical attributes of that subset (all the rows with value X in null).
From time to time you can do a `.head()/.title()` or an `.iloc[X:Y]` to check some things visually. But just as a "refresher".
Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook
#16I played around with many of these before: - https://github.com/quantopian/qgrid - https://github.com/man-group/dtale I find that I'm actually a lot faster using basic Pandas methods to get the data I want in exactly the form I want it. If I really want to show everything, I just use: ``` with pd.option_context('display.max_rows', None): print(df) ```
I use a similar function when I want to see everything:
```
def showAllRows(dataframeToShow):
with pd.option_context('display.max_rows', None, 'display.max_columns', None):
display(dataframeToShow)
showAllRows(df.head(1000))
```
Be warned though! if you call this function without limiting the number of rows to be fetched, it is guaranteed you will crash your machine. Always use head, sample or slices.
If do get a crush, then your only option is to open the ipynb file with vi and manually delete the millions of lines this function created.
Another function that I like is:
```
def showColumns(df, substring):
print([x for x in df.columns if substring in x])
return
showColumns(df, "year")
```
This is useful in data frames with many columns, when you want to find all the columns that have a specific string in their name. It returns a string, which then you can pass it in the dataframe to print only these columns.
Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook
#17Hey everyone. Mito cofounder here. Thanks to whoever posted this - was a real surprise to find it here :-)
Mito (pronounced my-toe) was born out of our personal experience with spreadsheets, and a previous (failed) spreadsheet version control product.
Spreadsheets were the original killer app for computers, and are the most popular programming language used worldwide today. That being said, spreadsheets have some growing to do! They don’t handle large datasets well, they don’t lead to repeatable or auditable processes, and generally they disrespect many of the hard won software engineering principals that us engineers fight for.
More than that, as spreadsheet users run into these problems and turn to Python to solve them, they struggle to use pandas to accomplish what would have been two clicks in a spreadsheet. Pandas is great, but the syntax is not always so obvious (not is learning to program in the first place!)
Mito is the our first step in addressing these problems. Take any dataframe, edit it like a spreadsheet, and generate code that corresponds to those edits. You can then take this Python code and use it in other scripts, send it to your colleagues, or just rerun it.
We’ve been working on Mito for over a year now. Growth has really picked up in the past few months - and we’ve begun working with larger companies to help accelerate their transition to Python.
To any companies who are somewhere in that Python transition process - please do reach out - we would love to see if we can be helpful for all your spreadsheet users!
Feel free to browse my profile for other spreadsheet related thoughts, I’m a bit of a HN junkie. Of course, any and all feedback (positive or negative) is appreciated.
My cofounders and I will be trolling about in the comments. Say hey! :-)
Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook
#18For those who are going through the thread finding new tools: pandas-profiling[0] is a library for automatic EDA (which bamboolib[1], mentioned elsewhere, also does).
[0]: https://github.com/pandas-profiling/pandas-profiling [1]: https://bamboolib.com/
Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook
#19Hey everyone. Mito cofounder here. Thanks to whoever posted this - was a real surprise to find it here :-) Mito (pronounced my-toe) was born out of our personal experience with spreadsheets, and a previous (failed) spreadsheet version control product. Spreadsheets were the original killer app for computers, and are the most popular programming language used worldwide today. That being said, spreadsheets have some gro…
Heyo! Another co-founder here. Excited to see Mito on HN :) Thanks @alefnula for posting!
+1 to everything @narush said.
It's important to us that the software we build is empowering to users and not restrictive. This plays out in two primary ways: 1) Since Mito is open source and generates Python code for every edit, Mito doesn't lock users into a 'Mito ecosystem', instead it help users interact with the powerful & robust Python ecosystem. 2) Because Mito is an extension to Jupyter Notebooks + JupyterLab, Mito improves your existing workflows instead of completely altering your data analytics stack.
Excited to interact with you all in the comments :)
Re: Mito – Excel-like interface for Pandas dataframes in Jupyter notebook
#20The telemetry thing is... weird. So we can use it for free but have no way to turn it off but upgrade to paid?
Thanks for that feedback. Mito's approach to telemetry is that we never log any of your data or metadata about your data. We don't track things like the size, shape, or content of your data.
We do collect info about app usage, things like which buttons users click. This allows us to focus development time on improving the features that are used most often.
That being said, it's important to us that there is a way to be totally telemetry-less if users don't want any information to be leave their computer. Compared to most other cloud-based sass data science tools where you pretty much have no hope of total privacy, we're proud of the flexibility that we offer.
But of course, we're always open to feedback about how we can continue to improve our practices!