surprising performance

[sthornington]

I am a little surprised at the performance of serde_arrow - a Vec<MyType> of only say 8 million records (~15 fields) is nevertheless taking 6+ seconds in to_record_batch -- is it possible that the generated code is not allocating the arrays with the total capacity to begin with or something ?

[chmp]

There are two issues:

• yes, the arrays are not pre-allocated. It did not seem to make a huge difference in early testing, but would be easy to add.
• serde arrow is using arrow to interpret the types, therefore there are lots of string look up involved. I would expect it take about 5 times longer, than if you would create the arrays manually

[sthornington]

Interesting okay thanks, where is the preallocation spot? I could test it out on my data.

[chmp]

My attempt was here: https://github.com/chmp/serde_arrow/pull/159/files -- but I never merged it. But looking it over, there also seems to a bug for nested sequences. It's also a rather small change. Maybe worthwhile to just add it.

[sthornington]

Thanks I'll take a look when I'm back at home in a couple of weeks. I'm not sure I understand the string lookups angle, surely once the tracing is done and the arrays are created there's no further need to lookup strings on O(n) of vec length?

[chmp]

The issue is that this package uses serde to understand Rust objects. For example using the expand macros tool in rust playground, for this struct

#[derive(serde::Serialize)]
struct Struct {
    a: i32,
    b: bool,
}

you get this generated code

#[doc(hidden)]
#[allow(non_upper_case_globals, unused_attributes, unused_qualifications, clippy :: absolute_paths,)]
const _: () =
    {
        #[allow(unused_extern_crates, clippy :: useless_attribute)]
        extern crate serde as _serde;
        #[automatically_derived]
        impl _serde::Serialize for Struct {
            fn serialize<__S>(&self, __serializer: __S) -> _serde::__private::Result<__S::Ok, __S::Error> where __S: _serde::Serializer {
                let mut __serde_state =_serde::Serializer::serialize_struct(__serializer, "Struct", false as usize + 1 + 1)?;
                _serde::ser::SerializeStruct::serialize_field(&mut __serde_state, "a", &self.a)?;
                _serde::ser::SerializeStruct::serialize_field(&mut __serde_state, "b", &self.b)?;
                _serde::ser::SerializeStruct::end(__serde_state)
            }
        }
    };

The serialize_field call map to this call in serde_arrow's builder. There you find the lookup. It has specialized code for handling fields in the same order, but it still requires a lookup. TBH. never tested it explicitly, but as long as the field order does not change, it should be O(1) not O(n) per lookup. The comparison uses the fact that struct field names in serde are static and simply compares pointer and length. If you are using maps for your records, the lookup needs to happen dynamically. Here a BTreeMap is used, so I would expect O(log(n)) per lookup. Edit: It's O(n) as per the rust docs.

Impls:

• for struct fields
• for map fields

Addition: without knowing your exact use case. If you are using #[serde(flatten)], Serde will implicitly convert use struct to a map. This will massively impact performance.

Derive output for flatten

#[derive(serde::Serialize)]
struct Struct {
    a: i32,
    b: bool,
    #[serde(flatten)]
    inner: Inner,
}

#[derive(serde::Serialize)]
struct Inner {
    c: i8,
}

Output:

#[doc(hidden)]
#[allow(non_upper_case_globals, unused_attributes, unused_qualifications, clippy :: absolute_paths,)]
const _: () =
    {
        #[allow(unused_extern_crates, clippy :: useless_attribute)]
        extern crate serde as _serde;
        #[automatically_derived]
        impl _serde::Serialize for Struct {
            fn serialize<__S>(&self, __serializer: __S) -> _serde::__private::Result<__S::Ok, __S::Error> where __S: _serde::Serializer {
                let mut __serde_state =_serde::Serializer::serialize_map(__serializer, _serde::__private::None)?;
                _serde::ser::SerializeMap::serialize_entry(&mut __serde_state, "a", &self.a)?;
                _serde::ser::SerializeMap::serialize_entry(&mut __serde_state, "b", &self.b)?;
                _serde::Serialize::serialize(&&self.inner, _serde::__private::ser::FlatMapSerializer(&mut __serde_state))?;
                _serde::ser::SerializeMap::end(__serde_state)
            }
        }
    };

[chmp]

Started to look into this issue in #284