Interplanetary telemetry compression

We are using sqlite for some things. But for large quantities of simply structured telemetry data it is not very good. About as good as raw json. The reason it is not very efficient is that the disk layout is optimised for fast access.

Re: Interplanetary telemetry compression

#5

Post by rklaehn » Thu, Jun 21, 2018, 7:47 PM UTC

zrav wrote: post #2

Sounds like a use case for zstd with custom dictionary.

Yes, zstd is definitely on the todo list. The increased performance should be quite welcome on low power edge devices.

I tried to get it to work, but had some trouble with the existing javascript bindings. I have only recently started developing in the javascript ecosystem.

PRs welcome...

Re: Interplanetary telemetry compression

#6

Post by planteen » Thu, Jun 21, 2018, 8:45 PM UTC

Did the telemetry arrive in CCSDS format? Did you consider directly logging all of the CCSDS headers?

Re: Interplanetary telemetry compression

#7

Post by rklaehn » Thu, Jun 21, 2018, 8:59 PM UTC

planteen wrote: post #6

Did the telemetry arrive in CCSDS format? Did you consider directly logging all of the CCSDS headers?

We were storing telemetry that was preprocessed by the MCS. This is preferable for analysis, since you can not afford to pipe the CCSDS packets through the rather slow mission control systems every time you want to plot or analyse the data. This also has the advantage that you store the data exactly as seen in the control room.

The raw CCSDS packet stream also gets stored, but given that the MCS systems are rather inflexible, they are not as valuable for general analysis.

Re: Interplanetary telemetry compression

#8

Post by danielvf » Thu, Jun 21, 2018, 9:37 PM UTC

What IPFS software are you using?

I've seen some bugs and issues with the official go server, and didn't feel it was production ready.

Re: Interplanetary telemetry compression

#9

Post by rklaehn » Thu, Jun 21, 2018, 9:52 PM UTC

danielvf wrote: post #8

What IPFS software are you using? I've seen some bugs and issues with the official go server, and didn't feel it was production ready.

0.4.15 currently. There are some things that are not production ready (in particular IPNS), but the basic infrastructure (the distributed content-addressed storage) is pretty solid. Discovery also works pretty well. NAT hole punching is impressive. Resource usage is low enough that you can run it on a raspberry pi with some room to spare.

For IPNS I am using the DNS txt record workaround until IPNS gets more stable. See http://blog.klaehn.org/2018/06/06/publish-blog-on-ipfs/ for how that works.

I am publishing the blog and several static pages on ipfs. So far it works really well.

Re: Interplanetary telemetry compression

#10

Post by seppel » Thu, Jun 21, 2018, 10:15 PM UTC

rklaehn wrote: post #5

zrav wrote: post #2

Sounds like a use case for zstd with custom dictionary.

Yes, zstd is definitely on the todo list. The increased performance should be quite welcome on low power edge devices. I tried to get it to work, but had some trouble with the existing javascript bindings. I have only recently started developing in the javascript ecosystem. PRs welcome...

It's not only increased performance. Since you can precompute a dictionary with zstd, you'll (most likely) get much better compression. Or you can at least stick to smaller block sizes.