Skip to content

Basic Phoenix salt bucket functionality added #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
eriknor wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Basic Phoenix salt bucket functionality added #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3588187
Added salt functionality to Phoenix table creation DDL and properties…
Mar 23, 2016
cbceea0
Fixed README.md for tablename salt
Mar 23, 2016
8132709
fix map get handling
Mar 23, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,12 @@ typeMap exists because Hive types and Phoenix types are not 1 to 1. "string|varc

The "jars" property allows end-users to supply a comma separated list of jars which need to be available on the classpath of the executors (JVM libraries, etc)

***Tables can be salted using the two salt properties;***

DST_TABLE_NAMESalt is a number declaring the number of salt buckets to use for the table, the tablename must be all uppercased

salt is a number declaring the number of salt buckets to use for any table not listed in the above property (default salt buckets)

***Saving Phoenix tables into Hive:***

To query Phoenix tables and save them back to Hive, change "destination=phoenix" to "destination=hive". When saving tables to Hive, "format" specifies the intended file-format (ORC, Parquet, Avro, text, etc).
Expand Down
5 changes: 5 additions & 0 deletions src/main/scala/com/github/randerzander/HiveToPhoenix.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,11 @@ object HiveToPhoenix{
command += field.name + " " + dstType + ","
}
command += " constraint my_pk primary key ("+pk+"))"
if (props.contains(dstTables(i)+"Salt")){
command += " SALT_BUCKETS = "+props.getOrElse(dstTables(i)+"Salt","10");
}else if(props.contains("salt")){
command += " SALT_BUCKETS = "+props.getOrElse("salt","10");
}
println("INFO: DESTINATION DDL:\n" + command)
// Execute Phoenix DDL
getConn(jdbcClass, connStr).createStatement().execute(command)
Expand Down