Using beam as a library, or at least allow imporing its packages

[geoah]

Unless I'm missing something (a canonical package name maybe?), beam's structure makes it really really hard to go get any of its packages.

Ideally I'd be really interested in being able to use beam as a library in another service, skipping the gpc server part, clustering, and optionally even rocksdb.

But even that's not possible, being able to import and use util/grpc/client to connect to a beam server would be a big improvement to having to re-generate the grpc stuff in the client service.

---

Would there be any chances of getting any of these or accepting PRs for them or other ways of getting to the end goal (other than keeping our own forks)?

• Move the beam packages to the top level
• ^ or introduce a canonical url we can import that points to the nested directory.
• Get the protoc, genny, and other generated files commited.
• Replace the custom dep tool with go mod (sarama needs to use the new package, and cheggaaa/pb needs to have their mod fix pr merged and the new version imported and everything else seems fine).

---

ps This is an amazing project and bql feels very nice and easy to use, thank you for releasing this publicly. :)

[superfell]

I had a pass at moving to go modules about a month ago. The go side of its was mostly straight forward (as you note, there are a few libraries that have go mod issues). The biggest problem was around the include path for the .proto generator. The proto dependencies end up in the module tree, which have version info added to the directory names, so you can't use the module tree directly for the protos (like it does today with the vendor tree) and using the vendoring tool, it only vendors go code.

Am happy to move to modules if the proto compile issues can be resolved. If we go this route, then we might also want to shuffle some other stuff around, as you end up with a top level directory that is a fairly random mix of go packages and other stuff.

[superfell]

This would also address the godoc.org issues mentioned in #21

[geoah]

@superfell thanks for a very quick response.

I see the issue with the proto generation but you should be able to decouple developing for the package from using the package.

People who use the package won't care about the actual proto files or their dependencies, so they don't even need to have protoc installed, or the proto deps fetched. This is why I think committing the generated files is important for allowing beam to be importable.

People who want to develop on beam can still use a tool such as your custom deps that uses both go mod vendor for the go deps, and git clone ... ./vendor/... for the proto deps. From there the makefile or build tool could opt for using the -mod=vendor flag or GOFLAGS=-mod=vendor env var to use the local vendor folder.

In regards to the shuffling, you could keep most things where they are and move the code to something closer to golang-standards/project-layout.

• current tools/* would be split between /cmd and /tools
• current utils/*, grpc handlers, other things you don't want people to touch would move to /internal
• current viewclient would probably be under /pkg to allow others to import it (same with hopefully the outer packages required for others to use the graph db as a lib)

I understand this is not an easy nor straightforward thing to fix and I really want to thank you for taking the time. :)

[shanghai-Jerry]

i am still confused about how to deploy my own cluster beams, no time to think about the project structure about it. Anyone can tell me how to deploy prod beam server to use. Any Docs?