Arguing against using protobuffers

Post by Vendan » Wed, Oct 10, 2018, 10:56 PM UTC

atombender wrote: post #97

Vendan wrote: post #90

Earlier quoted context omitted.

so, just for fun, here's a way to do that check without double unmarshalling and allocating maps and such for everything under your struct (also does a check for extra fields, but you could pull that out if you want): type Post struct { Title string `json:"title"` } func (v *Post) UnmarshalJSON(b []byte) error { dec := json.NewDecoder(bytes.NewReader(b)) required := map[string]struct{}{ "title": struct{}{}, } tok, er…

Thanks, that's neat! The challenge here is that I also need to validate values (e.g. support minimum/maximum), not just within structs, but also standalone values, which means an UnmarshalJSON directly on the type. I might end up doing something like your example, though. (Rejecting extra fields is on my list!)

    package main
    
    import (
    	"bytes"
    	"encoding/json"
    	"errors"
    	"fmt"
    )
    
    func main() {
    	fmt.Println("Hello, playground")
    	var p Post
    	err := json.Unmarshal([]byte(`{"title": "test"}`), &p)
    	fmt.Println(err, p)
    	err = json.Unmarshal([]byte(`{"title2": "test"}`), &p)
    	fmt.Println(err, p)
    	err = json.Unmarshal([]byte(`{}`), &p)
    	fmt.Println(err, p)
    	err = json.Unmarshal([]byte(`{"title": "long test"}`), &p)
    	fmt.Println(err, p)
    }
    
    type Post struct {
    	Title ShortTitle `json:"title"`
    }
    
    func (v *Post) UnmarshalJSON(b []byte) error {
    	dec := json.NewDecoder(bytes.NewReader(b))
    	required := map[string]struct{}{
    		"title": struct{}{},
    	}
    	tok, err := dec.Token()
    	if err != nil {
    		return err
    	}
    	if d, ok := tok.(json.Delim); !ok || d != '{' {
    		return errors.New("Expected object")
    	}
    	for {
    		tok, err := dec.Token()
    		if err != nil {
    			return err
    		}
    		if d, ok := tok.(json.Delim); ok && d == '}' {
    			break
    		}
    		switch tok {
    		case "title":
    			delete(required, "title")
    			err := dec.Decode(&v.Title)
    			if err != nil {
    				return err
    			}
    		default:
    			(*v) = Post{}
    			return errors.New(fmt.Sprintf("Unexpected field %s", tok))
    		}
    	}

    	if len(required) > 0 {
    		(*v) = Post{}
    		return errors.New(fmt.Sprintf("Missing %v required fields", len(required)))
    	}
    	return nil
    }
    
    type ShortTitle string
    
    func (st *ShortTitle) UnmarshalJSON(b []byte) error {
    	var tmpst string
    	err := json.Unmarshal(b, &tmpst)
    	if err != nil {
    		return err
    	}
    	if len(tmpst) > 6 {
    		return errors.New("Title too long!")
    	}
    	*st = ShortTitle(tmpst)
    	return nil
    }

Post by lowmagnet » Wed, Oct 10, 2018, 10:59 PM UTC

dweis wrote: post #60

Hi there, I'm an actual author of Protocol Buffers :) I think Sandy's analysis would benefit from considering why Protocol Buffers behave the way they do rather than outright attacking the design because it doesn't appear to make sense from a PL-centric perspective. As with all software systems, there are a number of competing constraints that have been weighed that have led to compromises. - D P.S. I also don't beli…

Post by amluto » Wed, Oct 10, 2018, 11:01 PM UTC

tathougies wrote: post #84

amluto wrote: post #63

Earlier quoted context omitted.

This may be easy, but it’s wrong . Endianness issues are just the start. Information leaks due to padding are a big deal. And it straight up doesn’t work if nontrivial data structures are involved.

Endianness issues are something every programmer should be aware of when sending data over the wire. I'm sorry I didn't insert the htons, htonls in my C code. In the Haskell code, endianness is handled by your Storable implementation (you define it after all, and can customize it however you want). I agree my example is somewhat tongue in cheek. The point though is that most languages have standard libraries for deal…

Post by tathougies » Wed, Oct 10, 2018, 11:06 PM UTC

amluto wrote: post #106

tathougies wrote: post #84

Earlier quoted context omitted.

Endianness issues are something every programmer should be aware of when sending data over the wire. I'm sorry I didn't insert the htons, htonls in my C code. In the Haskell code, endianness is handled by your Storable implementation (you define it after all, and can customize it however you want). I agree my example is somewhat tongue in cheek. The point though is that most languages have standard libraries for deal…

htons, etc are very 1980s, and I’ll make a fairly strong claim: they should never be used in new code, with a single exception. The reason is that an int with network endianness simply should not exist. In other words, when someone sends you a four byte big-endian integer, they sent four bytes, not an int. You can turn it into an int by shifting each byte by the relevant amount and oring them together. And a modern c…

Post by _fq4v » Wed, Oct 10, 2018, 11:07 PM UTC

romed wrote: post #105

_fq4v wrote: post #93

Earlier quoted context omitted.

C++ is a legacy language when it comes to data representation. Its ideas on what constitutes 'data' are from a different era. That is not a criticism of C++ or its utility. It is just a fact that data representation has changed a lot since the time of C++s development. Products and coproduct types are not academic wankery.

The machine does not think about category theory. The machine thinks about numbers. It can add them together! The way the machine thinks about numbers has not meaningfully changed in 40+ years.