Earlier quoted context omitted.
If you put an apostrophe in front of the number it will store it as text but will appear as a number. Just to be clear - you can then save the xls/xlsx to csv and the csv won't contain the leading apostrophe, but will show the whole number.
If you put the apostrophe in front of a number, and save it as csv, that works. But if you reopen that exact same file, Excel will still misinterpret the number. The issue here was that there's no way to let Excel know that this value in this CSV file is not a number. The only way around it that I know of is to stick with xlsx, which has its own pain points.
Another way is to rename the .csv file to be .txt instead, and load it in normally. This triggers the Text Import Wizard upon opening, and the rest is as above.
Also to do this programmatically, one can use the VBA function Workbook.OpenText, and specify the data types via the FieldInfo parameter.