orc stripe size in native writer

[wankunde]

Hi, in https://github.com/bytedance/bolt/blob/main/bolt/dwio/orc/writer/OrcWriter.cpp#L198, arrowOption->stripe_size = options.dataPageSize; ,

but stripe_size is 64MB by default in arrow and the orc java code while dataPageSize is int64_t dataPageSize = 1'024 * 1'024; in https://github.com/bytedance/bolt/blob/main/bolt/dwio/orc/writer/OrcWriter.h#L131

// struct ARROW_EXPORT WriteOption in arrow/adapters/orc/options.h
  /// Size of each ORC stripe in bytes, default 64 MiB
  int64_t stripe_size = 64 * 1024 * 1024;

// OrcConf.java

STRIPE_SIZE("orc.stripe.size", "hive.exec.orc.default.stripe.size",
      64L * 1024 * 1024,
      "Define the default ORC stripe size, in bytes."),
  

I don't understand why the default stripe size is changed from 64MB to 1MB ?
Thanks

[yangzhg]

Thanks for bringing this up.
To be honest, we haven't use ORC at a massive scale in production yet, so this default value might be conservative.
Could you share if you encountered any specific issues with the 1MB setting? (e.g., poor compression ratio or too many small I/Os?)
If the current default is causing performance degradation on your side, we are more than happy to align it with the community standard (64MB).