Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Spotting base64 encoded JSON, certificates, and private keys

ergaster.org

Post reply on HN
Print view View original

121–130 of 135 posts

Re: Spotting base64 encoded JSON, certificates, and private keys

#121

Post by 0points »

nneonneo wrote: post #96

Useful ones to know: - `R0lGOD` - GIF files - `iVBOR` - PNG files - `/9j/` - JPG files - `eyJ` - JSON - `PD94` - XML - `MII` - ASN.1 file, such as a certificate or private key These are nice to know since they show up pretty frequently (images in data: URLs, JSON/XML/ASN.1 in various protocols).

> GIF, PNG, JPG

It makes more sense to transmit binary formats in binary.

You would save bandwidth, memory and a decoding step.

Then you could also inspect the header bytes, instead of memorizing how they present in some intermediate encoding.

Re: Spotting base64 encoded JSON, certificates, and private keys

#122

Post by demurgos »

0points wrote: post #121
nneonneo wrote: post #96

Useful ones to know: - `R0lGOD` - GIF files - `iVBOR` - PNG files - `/9j/` - JPG files - `eyJ` - JSON - `PD94` - XML - `MII` - ASN.1 file, such as a certificate or private key These are nice to know since they show up pretty frequently (images in data: URLs, JSON/XML/ASN.1 in various protocols).

> GIF, PNG, JPG It makes more sense to transmit binary formats in binary. You would save bandwidth, memory and a decoding step. Then you could also inspect the header bytes, instead of memorizing how they present in some intermediate encoding.

Sometimes you need to embed binary data in a text format (e.g. JSON).

Re: Spotting base64 encoded JSON, certificates, and private keys

#123

Post by naikrovek »

demurgos wrote: post #122
0points wrote: post #121

Earlier quoted context omitted.

> GIF, PNG, JPG It makes more sense to transmit binary formats in binary. You would save bandwidth, memory and a decoding step. Then you could also inspect the header bytes, instead of memorizing how they present in some intermediate encoding.

Sometimes you need to embed binary data in a text format (e.g. JSON).

the amount of data i have stuffed into json as base64 encoded text makes me sick.

i wrote a glTF model converter once. 99% of those millions of JSON files I wrote were base64 encoded binary data.

a single glTF model sometimes wants to be two files on disk. one for the JSON and one for the binary data, and you use the JSON to describe where in the binary data the vertices are defined, and other windows for the various other bits like the triangles, triangle fans, textures, and other stuff are stored. But you can also base64 encode that data and put it in the JSON file and not have a messy double-file model. so that's what I did and I hated it. but it still felt better than having .gltf files and .bin files which together made up a single model file.

Re: Spotting base64 encoded JSON, certificates, and private keys

#124

Post by naikrovek »

derefr wrote: post #43
cogman10 wrote: post #8

I don't really love this. It just feels so wasteful. JWT does it as well. Even in this example, they are double base64 encoding strings (the salt). It's really too bad that there's really nothing quite like json. Everything speaks it and can write it. It'd be nice if something like protobuf was easier to write and read in a schemeless fashion.

> It'd be nice if something like protobuf was easier to write and read in a schemeless fashion. If you just want a generic, binary, hierarchical type-length-value encoding, have you considered https://en.wikipedia.org/wiki/Interchange_File_Format ? It's not that there are widely-supported IFF libraries, per se; but rather that the format is so simple that as long as your language has a byte-array type, you can code a…

yeah! I agree with this. I use plain TLV (which is very close to this IFF format) and is similar to how PNG stores all its chunks in a single file. As you mentioned.

I got grief for saying that I prefer TLV data over textual data (even if the data is text) because of how easy it is to write code to output and ingest this format, and it is way, WAY faster than JSON will ever be.

It really is a very easy way to get much faster transmission of data over the wire than JSON, and it's dead easy to write viewers for. It's just an underrated way to store binary data. storing things as binary is underrated in general.

Re: Spotting base64 encoded JSON, certificates, and private keys

#126

Post by mr_mitm »

0points wrote: post #121
nneonneo wrote: post #96

Useful ones to know: - `R0lGOD` - GIF files - `iVBOR` - PNG files - `/9j/` - JPG files - `eyJ` - JSON - `PD94` - XML - `MII` - ASN.1 file, such as a certificate or private key These are nice to know since they show up pretty frequently (images in data: URLs, JSON/XML/ASN.1 in various protocols).

> GIF, PNG, JPG It makes more sense to transmit binary formats in binary. You would save bandwidth, memory and a decoding step. Then you could also inspect the header bytes, instead of memorizing how they present in some intermediate encoding.

Knowing these magic bytes in base64 is mostly relevant in situations in which you see data encoded by other people, which means you probably had no control over the encoding. Other people (or rather every body) sometimes do things which don't make sense.

Re: Spotting base64 encoded JSON, certificates, and private keys

#127

Post by throwaway4496 »

fc417fc802 wrote: post #110
creatonez wrote: post #102

Earlier quoted context omitted.

They were probably expecting base64 encoded binary data. Base64-encoded-binary-inside-Base64-encoded-JSON-inside-JSON is a really strange construction if you haven't encountered it before, because of how much space it's wasting playing a game of Russian nesting dolls.

Just add a layer of compression periodically to reclaim the wasted space and it will all work out.

If you have a compression that works on encrypted data, you can avoid wasting your time on the "encryption".

Re: Spotting base64 encoded JSON, certificates, and private keys

#128

Post by moralestapia »

0points wrote: post #121
nneonneo wrote: post #96

Useful ones to know: - `R0lGOD` - GIF files - `iVBOR` - PNG files - `/9j/` - JPG files - `eyJ` - JSON - `PD94` - XML - `MII` - ASN.1 file, such as a certificate or private key These are nice to know since they show up pretty frequently (images in data: URLs, JSON/XML/ASN.1 in various protocols).

> GIF, PNG, JPG It makes more sense to transmit binary formats in binary. You would save bandwidth, memory and a decoding step. Then you could also inspect the header bytes, instead of memorizing how they present in some intermediate encoding.

Thanks, GPT.

Re: Spotting base64 encoded JSON, certificates, and private keys

#129

Post by fc417fc802 »

throwaway4496 wrote: post #127
fc417fc802 wrote: post #110

Earlier quoted context omitted.

Just add a layer of compression periodically to reclaim the wasted space and it will all work out.

If you have a compression that works on encrypted data, you can avoid wasting your time on the "encryption".

Base64 isn't encryption. The overhead added follows an extremely predictable pattern. That said I've no idea what the performance of common compression algorithms might be in such a use case. The comment was entirely tongue in cheek.

Re: Spotting base64 encoded JSON, certificates, and private keys

#130

Post by ThrowawayTestr »

0points wrote: post #121
nneonneo wrote: post #96

Useful ones to know: - `R0lGOD` - GIF files - `iVBOR` - PNG files - `/9j/` - JPG files - `eyJ` - JSON - `PD94` - XML - `MII` - ASN.1 file, such as a certificate or private key These are nice to know since they show up pretty frequently (images in data: URLs, JSON/XML/ASN.1 in various protocols).

> GIF, PNG, JPG It makes more sense to transmit binary formats in binary. You would save bandwidth, memory and a decoding step. Then you could also inspect the header bytes, instead of memorizing how they present in some intermediate encoding.

I have a tampermonkey script that's a megabyte in size because it includes an encoded gif file.
Post reply on HN