Welcome to vortex-java Discussions!

[dfa1]

👋 Welcome to vortex‑java Discussions

This space is for anyone interested in the Vortex columnar format, pure‑Java systems programming, or FFM‑based zero‑copy data processing.

Use Discussions to:
Ask questions about the implementation, design choices, or the Vortex spec
Share ideas for improvements, integrations, or performance experiments
Discuss architecture (FFM API, memory layout, encodings, vectorization)
Propose features or explore use cases
Collaborate with others building JVM‑native data infrastructure
Whether you're experimenting with Vortex, benchmarking, or exploring columnar formats, you’re welcome here.

👉 To get started
Introduce yourself below and tell us:

• What brought you to vortex‑java
• What you’re building or experimenting with
• Any questions or ideas you want to explore

[wpunit13]

Hello,

My job is in distributed/parallel processing engines. I have written a caching wrapper and few solutions which use either Arrow or parquet as serialization format. I have been experimenting with vortex on and off for 6 months. And I feel its now in shape where my pet projects can go and fully evaluate move from parquet to Vortex.

I did not want to go JNI route for few obvious reasons such as JNI Jar size is ~75mb, classloader conflicts, security + sandboxing I could go on and on. Then I thought of writing a short implementation of Java reader, and stumbled upon your good work.

Honestly big thanks for building this and going FFM. I will use this in my benchmark/evaluation. I would like to contribute as much as I can.

[dfa1]

Hello and welcome @wpunit13!

Thanks for your interest: in this repository there are several ways to import Parquet files (using the excellent Hardwood library by @gunnarmorling). Benchmarks against Parquet and Arrow from your workloads would be especially valuable because of my limited knowledge in those ecosystems, and so would any issues or gaps you hit on the way.

Contributions are welcome: Feel free to open issues or Pls anytime! Lately I tried to restructure the documentation around contributions: if you find any gaps, let me know!

PS
One small clarification: vortex-java it's a 100% pure Java implementation of the Vortex format, so there is no JNI, no native library and no Panama downcalls involved. The jars are split by reader/writer/tool roles and they are all below 1MB. vortexjni is used only for integration tests (that are mostly "we read data written by them", "them read data written by us").

[wpunit13]

Hey,

Thanks. I am aware of your implementation and thats exactly why I want to use this to read / write vortex. I did not want to use JNI to directly read write Vortex.

I will go through codebase and documentation. Once I am done with my experiments, I will get back to you on how I used vortex-java and benefits I got.