Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Excel.vim

github.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: Excel.vim

#41

Post by deadweight3 »

pyre wrote: post #17
deadweight3 wrote: post #8

Earlier quoted context omitted.

I hate this sentiment. I speak English and French and have limited amount of time to hack on software I'm giving away for free. My day job consists entirely in English. I have neither the experience, nor the inclination for internationalization of software. Don't like, don't use it. Or fork it, and add it yourself, because obviously you have more free time than I have.

> I have neither the experience, nor the inclination for internationalization of software. Taking a piece of software and making all of the UI language localized is one thing. Making sure that your program doesn't blow up if it encounters UTF-8 is another thing. Nowadays if your program chokes on UTF-8, I think it's safe to just consider it broken. In any case, looks like this is really where the issue may lie: # for…

Interesting definition of broken. It seems to work perfectly for me and the creator.

Re: Excel.vim

#42

Post by [deleted] »

deadweight3 wrote: post #41
pyre wrote: post #17

Earlier quoted context omitted.

> I have neither the experience, nor the inclination for internationalization of software. Taking a piece of software and making all of the UI language localized is one thing. Making sure that your program doesn't blow up if it encounters UTF-8 is another thing. Nowadays if your program chokes on UTF-8, I think it's safe to just consider it broken. In any case, looks like this is really where the issue may lie: # for…

Interesting definition of broken. It seems to work perfectly for me and the creator.

[deleted]

Re: Excel.vim

#43

Post by zzleeper »

sheetjs wrote: post #39

If the author is reading this: > For vim 7.3 and less, it works well for almost all kinds of file formats, > ie. .xls,.xlam,.xla,.xlsb,.xlsx,.xlsm,.xltx,.xltm,.xlt etc Someone already pointed it out ( https://github.com/yakiang/excel.vim/issues/5 ) on github: xlrd does not support the XLSB format (and the xlrd authors expressed no interest in building it)

I just found out about XLSB.. if really saves me so much time / space, I might just start using it for all large spreadsheets..

Re: Excel.vim

#44

Post by tracker1 »

rtpg wrote: post #10
deadweight3 wrote: post #8

Earlier quoted context omitted.

I hate this sentiment. I speak English and French and have limited amount of time to hack on software I'm giving away for free. My day job consists entirely in English. I have neither the experience, nor the inclination for internationalization of software. Don't like, don't use it. Or fork it, and add it yourself, because obviously you have more free time than I have.

the thing is if you build for unicode support from the start these conversations don't need to be had. The problem is not enough people treat text as a black box from the start (I can understand unwillingness to support bigger things like RTL)

Okay... let's go into this... how are the strings in excel encoded anyway?

I'd be willing to bet money that at least some of the formats in question aren't UTF-8, they are likely ASCII encoded against a character set or code page.

Then you have to read that codepage, and convert the necessary characters to their Unicode equivalents, and from there do you downcode to utf-8?

Does the language this library is written in support that translation? Are there modules to do that? Is the license for those module(s) necessary compatible?

Who's going to go through the different document versions to confirm, and adjust for the various encodings for non-ascii characters?

It's not as simple as saying "don't choke on unicode".

Re: Excel.vim

#45

Post by tracker1 »

sillysaurus3 wrote: post #25
a3n wrote: post #11

Earlier quoted context omitted.

There just isn't a lot of pervasive experience in the development community for multi-language unicode devlopment. Also xlrd is fairly old, although I don't know if that tool is part of what limits this to english. In ten years it might be better.

Joel Spolsky said that ten years ago. The problem is that devs are afraid to learn unicode. They treat it like learning a foreign language. It's not even a fun problem, like learning a new programming language, so nobody makes time for it. The only people who learn it are those who make it a point of pride to implement something correctly and handle corner cases. Unicode isn't even hard: Use UTF-8. Don't try to measu…

But these are file formats that may well not be encoded in UTF-8.. the formats already exist.. it isn't like he's creating a new spreadsheet format here. Some of them may well be encoded to something that works fine against unicode/utf-8, others not so much.

Re: Excel.vim

#46

Post by sheetjs »

zzleeper wrote: post #43
sheetjs wrote: post #39

If the author is reading this: > For vim 7.3 and less, it works well for almost all kinds of file formats, > ie. .xls,.xlam,.xla,.xlsb,.xlsx,.xlsm,.xltx,.xltm,.xlt etc Someone already pointed it out ( https://github.com/yakiang/excel.vim/issues/5 ) on github: xlrd does not support the XLSB format (and the xlrd authors expressed no interest in building it)

I just found out about XLSB.. if really saves me so much time / space, I might just start using it for all large spreadsheets..

XLSB is significantly faster to process because it does not require an XML parser to blast through the data. Numbers are stored directly as IEEE754 doubles. Text may end up larger because XLSB uses UTF16LE for everything, including pure-ASCII strings (since Excel uses that format internally, there is no conversion involved and XLSB still will be faster).

The problem you will encounter is that most programs (Numbers, Google Docs) do not support XLSB.

Shameless plug: https://github.com/SheetJS/js-xlsx supports both XLSX and XLSB (AFAICT the only liberally licensed project that handles the format)

Re: Excel.vim

#48

Post by sheetjs »

tracker1 wrote: post #44
rtpg wrote: post #10

Earlier quoted context omitted.

the thing is if you build for unicode support from the start these conversations don't need to be had. The problem is not enough people treat text as a black box from the start (I can understand unwillingness to support bigger things like RTL)

Okay... let's go into this... how are the strings in excel encoded anyway? I'd be willing to bet money that at least some of the formats in question aren't UTF-8, they are likely ASCII encoded against a character set or code page. Then you have to read that codepage, and convert the necessary characters to their Unicode equivalents, and from there do you downcode to utf-8? Does the language this library is written in…

> how are the strings in excel encoded anyway?

Length-prefixed byte arrays encoded using various code pages. There are a small number that excel uses: https://github.com/SheetJS/js-codepage/blob/master/excel.csv (the columns are CP#, mapping, single/double-byte)

> Does the language this library is written in support that translation? Are there modules to do that? Is the license for those module(s) necessary compatible?

If we can put together an Apache2-licensed module in JS in an afternoon (https://github.com/SheetJS/js-codepage) it can be done in python.

> Who's going to go through the different document versions to confirm, and adjust for the various encodings for non-ascii characters?

Someone already did that: https://github.com/SheetJS/test_files/tree/master/biff5 has artifacts for every language type

Re: Excel.vim

#49

Post by sillysaurus3 »

tracker1 wrote: post #45
sillysaurus3 wrote: post #25

Earlier quoted context omitted.

Joel Spolsky said that ten years ago. The problem is that devs are afraid to learn unicode. They treat it like learning a foreign language. It's not even a fun problem, like learning a new programming language, so nobody makes time for it. The only people who learn it are those who make it a point of pride to implement something correctly and handle corner cases. Unicode isn't even hard: Use UTF-8. Don't try to measu…

But these are file formats that may well not be encoded in UTF-8.. the formats already exist.. it isn't like he's creating a new spreadsheet format here. Some of them may well be encoded to something that works fine against unicode/utf-8, others not so much.

So you write FooToUTF8() and UTF8ToFoo(), where Foo is whatever the encoding is in the external format. Done.

As far as I know, UTF-8 will work 100% of the time, and is almost always the best internal representation for software you write due to how simple and uniform it is. If something is encoded in some other format, you can probably find a conversion function online.

Re: Excel.vim

#50

Post by tracker1 »

sillysaurus3 wrote: post #49
tracker1 wrote: post #45

Earlier quoted context omitted.

But these are file formats that may well not be encoded in UTF-8.. the formats already exist.. it isn't like he's creating a new spreadsheet format here. Some of them may well be encoded to something that works fine against unicode/utf-8, others not so much.

So you write FooToUTF8() and UTF8ToFoo(), where Foo is whatever the encoding is in the external format. Done. As far as I know, UTF-8 will work 100% of the time, and is almost always the best internal representation for software you write due to how simple and uniform it is. If something is encoded in some other format, you can probably find a conversion function online.

Okay, so why don't you fork the project, and create your simple Foo/UTF8 methods, and confirm that they are the correct Foo/UTF8 methods for each of the document formats supported.

I'm not saying that it's really all that hard, but there are multiple document formats, and versions of those formats. The author obviously didn't need unicode support, so didn't test for it. I'm sure test cases, and a pull request would be welcome.

Post reply on HN