<commit_before>// Copyright 2014 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

/*

Package design describes some of the data structures used in QL.

Handles

A handle is a 7 byte "pointer" to a block in the DB[0].

Scalar encoding

Encoding of so called "scalars" provided by [1]. Unless specified otherwise,
all values discussed below are scalars, encoded scalars or encoding of scalar
arrays.

Database root

DB root is found at a fixed handle (#1).

	+------+
	| head |
	+------+

Head is the handle of meta data for the first table or zero if there are no
tables in the DB..

Table meta data

Table meta data is a [4]scalar (a 4 field "record").

	+---+-------+--------+
	| # | Name  | Type   |
	+---+-------+--------+
	| 0 | next  | int64  |
	| 1 | scols | string |
	| 2 | hhead | int64  |
	| 3 | name  | string |
	+---+-------+--------+

The handle of the next table meta data is in the field #0 (next). If there is
no next table meta data, the field is zero. Names and types of table columns
are stored in field #1 (scols). A single field is described by concatenating a
type tag and the column name. The type tags are

	bool       'b'
	complex64  'c'
	complex128 'd'
	float32    'f'
	float64    'g', alias float
	int8       'i'
	int16      'j'
	int32      'k'
	int64      'l', alias int
	string     's'
	uint8      'u', alias byte
	uint16     'v'
	uint32     'w'
	uint64     'x', alias uint
	bigInt     'I'
	bigRat     'R'
	blob       'B'
	duration   'D'
	time       'T'

The scols value is the above described encoded fields joined using "|". For
example

	CREATE TABLE t (Foo bool, Bar string, Baz float);

This statement adds a table meta data with scols

	"bFool|sBar|gBaz"

Columns can be dropped from a table

	ALTER TABLE t DROP COLUMN Bar;

This "erases" the field info in scols, so the value becomes

	"bFool||gBaz"

Colums can be added to a table

	ALTER TABLE t ADD Count uint;

New fields are always added to the end of scols

	"bFool||gBaz|xCount"

Index of a field in strings.Split(scols, "|") is the index of the field in a
table record. The above discussed rules for column dropping and column adding
allow for schema evolution without a need to reshape any existing table data.
Dropped columns are left where they are and new records insert nil in their
place. The encoded nil is one byte. Added columns, when not present in
preexisting records are returned as nil values. If the overhead of dropped
columns becomes an issue and there's time/space and memory enough to move the
records of a table around:

	BEGIN TRANSACTION;
		CREATE TABLE new (column definitions);
		INSERT INTO new SELECT * FROM old;
		DROP TABLE old;
		CREATE TABLE old (column definitions);
		INSERT INTO old SELECT * FROM new;
		DROP TABLE new;
	END TRANSACTION;

This is not very time/space effective and for Big Data it can cause an OOM
because transactions are limited by memory resources available to the process.
Perhaps a method and/or QL statement to do this in-place should be added
(MAYBE).

Field #2 (hhead) is a handle to a head of table records, i.e. not a handle to
the first record in the table. It is thus always non zero even for a table
having no records. The reason for this "double pointer" schema is to enable
adding (linking) a new record by updating a single value of the (hhead pointing
to) head.

	tableMeta.hhead	-> head	-> firstTableRecord

Field #3 (name) is the table name.

Table record

A table record is an [N]scalar.

	+-----+------------+--------+------------------------------------+
	|  #  |    Name    |  Type  |                                    |
	+-----+------------+--------+------------------------------------+
	|  0  | next       | int64  | Handle of the next record or zero. |
	|  1  | id         | int64  | Automatically assigned unique      |
	|     |            |        | value obtainable by id().          |
	|  2  | field #0   | scalar | First field of the record.         |
	|  3  | field #1   | scalar | Second field of the record.        |
	     ...
	| N-1 | field #N-2 | scalar | Last field of the record.          |
	+-----+------------+--------+------------------------------------+

The linked "ordering" of table records has no semantics and it doesn't have to
correlate to the order of how the records were added to the table. In fact, an
efficient way of the linking leads to "ordering" which is actually reversed wrt
the insertion order.

Non scalar types

Scalar types of [1] are bool, complex*, float*, int*, uint*, string and []byte types. All
other types are "blob-like".

	QL type         Go type
	-----------------------------
	blob            []byte
	bigint          big.Int
	bigrat          big.Rat
	time            time.Time
	duration        time.Duration

Memory back-end stores the Go type directly. File back-end must resort to
encode all of the above as (tagged) []byte due to the lack of more types
supported natively by lldb. NULL values of blob-like types are encoded as nil
(gbNull in lldb/gb.go), exactly the same as the already existing QL types are.

Blob encoding

The values of the blob-like types are first encoded into a []byte slice:

	+-----------------------+-------------------+
	| blob                  | raw               |
	| bigint, bigrat, time	| gob encoded       |
	| duration		| gob encoded int64 |
	+-----------------------+-------------------+

The gob encoding is "differential" wrt an initial encoding of all of the
blob-like type. IOW, the initial type descriptors which gob encoding must write
out are stripped off and "resupplied" on decoding transparently. If the length
of the resulting slice is <= shortBlob, the first and only chunk
is the scalar encoding of

	[]interface{}{typeTag, slice}.

The length of slice can be zero (for blob("")). If the resulting slice is long
(> shortBlob), the first chunk comes from encoding

	[]interface{}{typeTag, nextHandle, firstPart}.

In this case len(firstPart) <= shortBlob. Second and other chunks: If the chunk
is the last one, src is

	[]interface{lastPart}.

In this case len(lastPart) <= 64kB. If the chunk is not the last one, src is

	[]interface{}{nextHandle, part}.

In this case len(part) == 64kB.

Links

Referenced from above:

  [0]: http://godoc.org/github.com/cznic/exp/lldb#hdr-Block_handles
  [1]: http://godoc.org/github.com/cznic/exp/lldb#EncodeScalars

Rationale

While these notes might be useful to anyone looking at QL sources, the
specifically intended reader is my future self.

*/
package design
<commit_msg>Consider indices in the design doc. Updates #14.<commit_after>// Copyright 2014 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

/*

Package design describes some of the data structures used in QL.

Handles

A handle is a 7 byte "pointer" to a block in the DB[0].

Scalar encoding

Encoding of so called "scalars" provided by [1]. Unless specified otherwise,
all values discussed below are scalars, encoded scalars or encoding of scalar
arrays.

Database root

DB root is a 1-scalar found at a fixed handle (#1).

	+---+------+--------+-----------------------+
	| # | Name |  Type  |     Description       |
	+---+------+--------+-----------------------+
	| 0 | head | handle | First table meta data |
	+---+------+--------+-----------------------+

Head is the handle of meta data for the first table or zero if there are no
tables in the DB..

Table meta data

Table meta data is a 6-scalar.

	+---+---------+--------+--------------------------+
	| # | Name    | Type   |      Description         |
	+---+---------+--------+--------------------------+
	| 0 | next    | handle | Table meta data.         |
	| 1 | scols   | string | Column defintitions      |
	| 2 | hhead   | handle | -> head -> first record  |
	| 3 | name    | string | Table name               |
	| 4 | indices | string | Index definitions        |
	| 5 | xroots  | handle | Index B+Trees roots list |
	+---+---------+--------+--------------------------+

Fields #4 and #5 are optional for backward compatibility with existing
databases.  OTOH, forward compatibility will not work. If the DB is created or
indices added using a newer version of QL which supports indexing then older
versions of QL, expecting only 4 fields of meta data will not be able to use
the DB. That's the intended behavior because the older versions of QL cannot
update the indexes, which can break queries runned by the newer QL version
which expect indices to be always actualized on any table-with-indices
mutation.

The handle of the next table meta data is in the field #0 (next). If there is
no next table meta data, the field is zero. Names and types of table columns
are stored in field #1 (scols). A single field is described by concatenating a
type tag and the column name. The type tags are

	bool       'b'
	complex64  'c'
	complex128 'd'
	float32    'f'
	float64    'g', alias float
	int8       'i'
	int16      'j'
	int32      'k'
	int64      'l', alias int
	string     's'
	uint8      'u', alias byte
	uint16     'v'
	uint32     'w'
	uint64     'x', alias uint
	bigInt     'I'
	bigRat     'R'
	blob       'B'
	duration   'D'
	time       'T'

The scols value is the above described encoded fields joined using "|". For
example

	CREATE TABLE t (Foo bool, Bar string, Baz float);

This statement adds a table meta data with scols

	"bFool|sBar|gBaz"

Columns can be dropped from a table

	ALTER TABLE t DROP COLUMN Bar;

This "erases" the field info in scols, so the value becomes

	"bFool||gBaz"

Colums can be added to a table

	ALTER TABLE t ADD Count uint;

New fields are always added to the end of scols

	"bFool||gBaz|xCount"

Index of a field in strings.Split(scols, "|") is the index of the field in a
table record. The above discussed rules for column dropping and column adding
allow for schema evolution without a need to reshape any existing table data.
Dropped columns are left where they are and new records insert nil in their
place. The encoded nil is one byte. Added columns, when not present in
preexisting records are returned as nil values. If the overhead of dropped
columns becomes an issue and there's time/space and memory enough to move the
records of a table around:

	BEGIN TRANSACTION;
		CREATE TABLE new (column definitions);
		INSERT INTO new SELECT * FROM old;
		DROP TABLE old;
		CREATE TABLE old (column definitions);
		INSERT INTO old SELECT * FROM new;
		DROP TABLE new;
	END TRANSACTION;

This is not very time/space effective and for Big Data it can cause an OOM
because transactions are limited by memory resources available to the process.
Perhaps a method and/or QL statement to do this in-place should be added
(MAYBE).

Field #2 (hhead) is a handle to a head of table records, i.e. not a handle to
the first record in the table. It is thus always non zero even for a table
having no records. The reason for this "double pointer" schema is to enable
adding (linking) a new record by updating a single value of the (hhead pointing
to) head.

	tableMeta.hhead	-> head	-> firstTableRecord

The table name is stored in field #3 (name).

Indices

Consider an index named N, indexing column named C. C has index X in scols.
The encoding of this particular index is a string "<tag>N". <tag> is a string
"n" for non unique indices and "u" for unique indices. There is this index
information for all columns of scols. Where the column is not indexed, the
index info is an empty string. Infos for all indexes are joined with "|". For
example

	BEGIN TRANSACTION;
		CREATE TABLE t (Foo int, Bar bool, Baz string);
		CREATE INDEX X ON t (Baz);
		CREATE UNIQUE INDEX Y ON t (Foo);
	COMMIT;

The values of fields #1 and #4 for the above is

	  scols: "lFoo|bBar|sBaz"
	indices: "uY||nX"

Note that there cannot be two differently named indexes for the same column and
it's intended. The indices are B+Trees[2]. The list of handles to their roots
is pointed to by xroots with zeros for non indexed columns. For the previous
example

	tableMeta.xroots -> {y, 0, x}

where x is the root of the B+Tree for the X index and y is the root of the
B+Tree for the Y index. Similarly to hhead, xroots is never zero, even when
there are no indices for a table.

Table record

A table record is an N-scalar.

	+-----+------------+--------+-------------------------------+
	|  #  |    Name    |  Type  |      Description              |
	+-----+------------+--------+-------------------------------+
	|  0  | next       | handle | Next record or zero.          |
	|  1  | id         | int64  | Automatically assigned unique |
	|     |            |        | value obtainable by id().     |
	|  2  | field #0   | scalar | First field of the record.    |
	|  3  | field #1   | scalar | Second field of the record.   |
	     ...
	| N-1 | field #N-2 | scalar | Last field of the record.     |
	+-----+------------+--------+-------------------------------+

The linked "ordering" of table records has no semantics and it doesn't have to
correlate to the order of how the records were added to the table. In fact, an
efficient way of the linking leads to "ordering" which is actually reversed wrt
the insertion order.

Non unique index

The index is a B+Tree[2]. The composite key of the B+Tree is {V, H}, where V is
the value of the indexed column of record with handle H. The B+Tree value is
not used.

Unique index

The index is a B+Tree[2]. If the indexed value V is not NULL then key of the
B+Tree key is V and the B+Tree value is H, where H is the handle of the indexed
record: V -> H. If the indexed value is NULL then the composite B+Tree key is
{nil, H} and the B+Tree value is not used.

Non scalar types

Scalar types of [1] are bool, complex*, float*, int*, uint*, string and []byte
types. All other types are "blob-like".

	QL type         Go type
	-----------------------------
	blob            []byte
	bigint          big.Int
	bigrat          big.Rat
	time            time.Time
	duration        time.Duration

Memory back-end stores the Go type directly. File back-end must resort to
encode all of the above as (tagged) []byte due to the lack of more types
supported natively by lldb. NULL values of blob-like types are encoded as nil
(gbNull in lldb/gb.go), exactly the same as the already existing QL types are.

Blob encoding

The values of the blob-like types are first encoded into a []byte slice:

	+-----------------------+-------------------+
	| blob                  | raw               |
	| bigint, bigrat, time	| gob encoded       |
	| duration		| gob encoded int64 |
	+-----------------------+-------------------+

The gob encoding is "differential" wrt an initial encoding of all of the
blob-like type. IOW, the initial type descriptors which gob encoding must write
out are stripped off and "resupplied" on decoding transparently. See also
blob.go. If the length of the resulting slice is <= shortBlob, the first and
only chunk is the scalar encoding of

	[]interface{}{typeTag, slice}.

The length of slice can be zero (for blob("")). If the resulting slice is long
(> shortBlob), the first chunk comes from encoding

	[]interface{}{typeTag, nextHandle, firstPart}.

In this case len(firstPart) <= shortBlob. Second and other chunks: If the chunk
is the last one, src is

	[]interface{lastPart}.

In this case len(lastPart) <= 64kB. If the chunk is not the last one, src is

	[]interface{}{nextHandle, part}.

In this case len(part) == 64kB.

Links

Referenced from above:

  [0]: http://godoc.org/github.com/cznic/exp/lldb#hdr-Block_handles
  [1]: http://godoc.org/github.com/cznic/exp/lldb#EncodeScalars
  [2]: http://godoc.org/github.com/cznic/exp/lldb#BTree

Rationale

While these notes might be useful to anyone looking at QL sources, the
specifically intended reader is my future self.

*/
package design
<|endoftext|>


<commit_before>/*
Copyright 2017 The Kubernetes Authors All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package cmd

import (
	"strings"

	"github.com/kubernetes/kompose/pkg/app"
	"github.com/kubernetes/kompose/pkg/kobject"
	log "github.com/sirupsen/logrus"
	"github.com/spf13/cobra"
	"github.com/spf13/viper"
)

// TODO: comment
var (
	ConvertOut                   string
	ConvertBuildRepo             string
	ConvertBuildBranch           string
	ConvertBuild                 string
	ConvertVolumes               string
	ConvertChart                 bool
	ConvertDeployment            bool
	ConvertDaemonSet             bool
	ConvertReplicationController bool
	ConvertYaml                  bool
	ConvertJSON                  bool
	ConvertStdout                bool
	ConvertEmptyVols             bool
	ConvertInsecureRepo          bool
	ConvertDeploymentConfig      bool
	ConvertReplicas              int
	ConvertController            string
	ConvertPushImage             bool
	ConvertOpt                   kobject.ConvertOptions
	ConvertYAMLIndent            int

	UpBuild string

	// WithKomposeAnnotation decides if we will add metadata about this convert to resource's annotation.
	// default is true.
	WithKomposeAnnotation bool

	// MultipleContainerMode which enables creating multi containers in a single pod is a developping function.
	// default is false
	MultipleContainerMode bool
)

var convertCmd = &cobra.Command{
	Use:   "convert [file]",
	Short: "Convert a Docker Compose file",
	PreRun: func(cmd *cobra.Command, args []string) {

		// Check that build-config wasn't passed in with --provider=kubernetes
		if GlobalProvider == "kubernetes" && UpBuild == "build-config" {
			log.Fatalf("build-config is not a valid --build parameter with provider Kubernetes")
		}

		// Create the Convert Options.
		ConvertOpt = kobject.ConvertOptions{
			ToStdout:                    ConvertStdout,
			CreateChart:                 ConvertChart,
			GenerateYaml:                ConvertYaml,
			GenerateJSON:                ConvertJSON,
			Replicas:                    ConvertReplicas,
			InputFiles:                  GlobalFiles,
			OutFile:                     ConvertOut,
			Provider:                    GlobalProvider,
			CreateD:                     ConvertDeployment,
			CreateDS:                    ConvertDaemonSet,
			CreateRC:                    ConvertReplicationController,
			Build:                       ConvertBuild,
			BuildRepo:                   ConvertBuildRepo,
			BuildBranch:                 ConvertBuildBranch,
			PushImage:                   ConvertPushImage,
			CreateDeploymentConfig:      ConvertDeploymentConfig,
			EmptyVols:                   ConvertEmptyVols,
			Volumes:                     ConvertVolumes,
			InsecureRepository:          ConvertInsecureRepo,
			IsDeploymentFlag:            cmd.Flags().Lookup("deployment").Changed,
			IsDaemonSetFlag:             cmd.Flags().Lookup("daemon-set").Changed,
			IsReplicationControllerFlag: cmd.Flags().Lookup("replication-controller").Changed,
			Controller:                  strings.ToLower(ConvertController),
			IsReplicaSetFlag:            cmd.Flags().Lookup("replicas").Changed,
			IsDeploymentConfigFlag:      cmd.Flags().Lookup("deployment-config").Changed,
			YAMLIndent:                  ConvertYAMLIndent,
			WithKomposeAnnotation:       WithKomposeAnnotation,
			MultipleContainerMode:       MultipleContainerMode,
		}

		// Validate before doing anything else. Use "bundle" if passed in.
		app.ValidateFlags(GlobalBundle, args, cmd, &ConvertOpt)
		app.ValidateComposeFile(&ConvertOpt)
	},
	Run: func(cmd *cobra.Command, args []string) {

		app.Convert(ConvertOpt)
	},
}

func init() {
	// Automatically grab environment variables
	viper.AutomaticEnv()

	// Kubernetes only
	convertCmd.Flags().BoolVarP(&ConvertChart, "chart", "c", false, "Create a Helm chart for converted objects")
	convertCmd.Flags().BoolVar(&ConvertDaemonSet, "daemon-set", false, "Generate a Kubernetes daemonset object (deprecated, use --controller instead)")
	convertCmd.Flags().BoolVarP(&ConvertDeployment, "deployment", "d", false, "Generate a Kubernetes deployment object (deprecated, use --controller instead)")
	convertCmd.Flags().BoolVar(&ConvertReplicationController, "replication-controller", false, "Generate a Kubernetes replication controller object (deprecated, use --controller instead)")
	convertCmd.Flags().StringVar(&ConvertController, "controller", "", `Set the output controller ("deployment"|"daemonSet"|"replicationController")`)
	convertCmd.Flags().MarkDeprecated("daemon-set", "use --controller")
	convertCmd.Flags().MarkDeprecated("deployment", "use --controller")
	convertCmd.Flags().MarkDeprecated("replication-controller", "use --controller")
	convertCmd.Flags().MarkHidden("chart")
	convertCmd.Flags().MarkHidden("daemon-set")
	convertCmd.Flags().MarkHidden("replication-controller")
	convertCmd.Flags().MarkHidden("deployment")
	convertCmd.Flags().BoolVar(&MultipleContainerMode, "multiple-container-mode", false, "Create multiple containers grouped by 'kompose.service.group' label")

	// OpenShift only
	convertCmd.Flags().BoolVar(&ConvertDeploymentConfig, "deployment-config", true, "Generate an OpenShift deploymentconfig object")
	convertCmd.Flags().BoolVar(&ConvertInsecureRepo, "insecure-repository", false, "Use an insecure Docker repository for OpenShift ImageStream")
	convertCmd.Flags().StringVar(&ConvertBuildRepo, "build-repo", "", "Specify source repository for buildconfig (default remote origin)")
	convertCmd.Flags().StringVar(&ConvertBuildBranch, "build-branch", "", "Specify repository branch to use for buildconfig (default master)")
	convertCmd.Flags().MarkDeprecated("deployment-config", "use --controller")
	convertCmd.Flags().MarkHidden("deployment-config")
	convertCmd.Flags().MarkHidden("insecure-repository")
	convertCmd.Flags().MarkHidden("build-repo")
	convertCmd.Flags().MarkHidden("build-branch")

	// Standard between the two
	convertCmd.Flags().StringVar(&ConvertBuild, "build", "none", `Set the type of build ("local"|"build-config"(OpenShift only)|"none")`)
	convertCmd.Flags().BoolVar(&ConvertPushImage, "push-image", true, "If we should push the docker image we built")
	convertCmd.Flags().BoolVarP(&ConvertYaml, "yaml", "y", false, "Generate resource files into YAML format")
	convertCmd.Flags().MarkDeprecated("yaml", "YAML is the default format now.")
	convertCmd.Flags().MarkShorthandDeprecated("y", "YAML is the default format now.")
	convertCmd.Flags().BoolVarP(&ConvertJSON, "json", "j", false, "Generate resource files into JSON format")
	convertCmd.Flags().BoolVar(&ConvertStdout, "stdout", false, "Print converted objects to stdout")
	convertCmd.Flags().StringVarP(&ConvertOut, "out", "o", "", "Specify a file name or directory to save objects to (if path does not exist, a file will be created)")
	convertCmd.Flags().IntVar(&ConvertReplicas, "replicas", 1, "Specify the number of replicas in the generated resource spec")
	convertCmd.Flags().StringVar(&ConvertVolumes, "volumes", "persistentVolumeClaim", `Volumes to be generated ("persistentVolumeClaim"|"emptyDir"|"hostPath" | "configMap")`)

	convertCmd.Flags().BoolVar(&WithKomposeAnnotation, "with-kompose-annotation", true, "Add kompose annotations to generated resource")

	// Deprecated commands
	convertCmd.Flags().BoolVar(&ConvertEmptyVols, "emptyvols", false, "Use Empty Volumes. Do not generate PVCs")
	convertCmd.Flags().MarkDeprecated("emptyvols", "emptyvols has been marked as deprecated. Use --volumes empty")

	convertCmd.Flags().IntVar(&ConvertYAMLIndent, "indent", 2, "Spaces length to indent generated yaml files")

	// In order to 'separate' both OpenShift and Kubernetes only flags. A custom help page is created
	customHelp := `Usage:{{if .Runnable}}
  {{if .HasAvailableFlags}}{{appendIfNotPresent .UseLine "[flags]"}}{{else}}{{.UseLine}}{{end}}{{end}}{{if .HasAvailableSubCommands}}
  {{ .CommandPath}} [command]{{end}}{{if gt .Aliases 0}}

Aliases:
  {{.NameAndAliases}}
{{end}}{{if .HasExample}}

Examples:
{{ .Example }}{{end}}{{ if .HasAvailableSubCommands}}
Available Commands:{{range .Commands}}{{if .IsAvailableCommand}}
  {{rpad .Name .NamePadding }} {{.Short}}{{end}}{{end}}{{end}}{{ if .HasAvailableLocalFlags}}

Kubernetes Flags:
      --daemon-set               Generate a Kubernetes daemonset object (deprecated, use --controller instead)
  -d, --deployment               Generate a Kubernetes deployment object (deprecated, use --controller instead)
  -c, --chart                    Create a Helm chart for converted objects
      --replication-controller   Generate a Kubernetes replication controller object (deprecated, use --controller instead)

OpenShift Flags:
      --build-branch             Specify repository branch to use for buildconfig (default is current branch name)
      --build-repo               Specify source repository for buildconfig (default is current branch's remote url)
      --deployment-config        Generate an OpenShift deployment config object
      --insecure-repository      Specify to use insecure docker repository while generating Openshift image stream object

Flags:
{{.LocalFlags.FlagUsages | trimRightSpace}}{{end}}{{ if .HasAvailableInheritedFlags}}

Global Flags:
{{.InheritedFlags.FlagUsages | trimRightSpace}}{{end}}{{if .HasHelpSubCommands}}

Additional help topics:{{range .Commands}}{{if .IsHelpCommand}}
  {{rpad .CommandPath .CommandPathPadding}} {{.Short}}{{end}}{{end}}{{end}}{{ if .HasAvailableSubCommands }}
Use "{{.CommandPath}} [command] --help" for more information about a command.{{end}}
`
	// Set the help template + add the command to root
	convertCmd.SetUsageTemplate(customHelp)

	RootCmd.AddCommand(convertCmd)
}
<commit_msg>fix: --build flag default change to local<commit_after>/*
Copyright 2017 The Kubernetes Authors All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package cmd

import (
	"strings"

	"github.com/kubernetes/kompose/pkg/app"
	"github.com/kubernetes/kompose/pkg/kobject"
	log "github.com/sirupsen/logrus"
	"github.com/spf13/cobra"
	"github.com/spf13/viper"
)

// TODO: comment
var (
	ConvertOut                   string
	ConvertBuildRepo             string
	ConvertBuildBranch           string
	ConvertBuild                 string
	ConvertVolumes               string
	ConvertChart                 bool
	ConvertDeployment            bool
	ConvertDaemonSet             bool
	ConvertReplicationController bool
	ConvertYaml                  bool
	ConvertJSON                  bool
	ConvertStdout                bool
	ConvertEmptyVols             bool
	ConvertInsecureRepo          bool
	ConvertDeploymentConfig      bool
	ConvertReplicas              int
	ConvertController            string
	ConvertPushImage             bool
	ConvertOpt                   kobject.ConvertOptions
	ConvertYAMLIndent            int

	UpBuild string

	// WithKomposeAnnotation decides if we will add metadata about this convert to resource's annotation.
	// default is true.
	WithKomposeAnnotation bool

	// MultipleContainerMode which enables creating multi containers in a single pod is a developping function.
	// default is false
	MultipleContainerMode bool
)

var convertCmd = &cobra.Command{
	Use:   "convert [file]",
	Short: "Convert a Docker Compose file",
	PreRun: func(cmd *cobra.Command, args []string) {

		// Check that build-config wasn't passed in with --provider=kubernetes
		if GlobalProvider == "kubernetes" && UpBuild == "build-config" {
			log.Fatalf("build-config is not a valid --build parameter with provider Kubernetes")
		}

		// Create the Convert Options.
		ConvertOpt = kobject.ConvertOptions{
			ToStdout:                    ConvertStdout,
			CreateChart:                 ConvertChart,
			GenerateYaml:                ConvertYaml,
			GenerateJSON:                ConvertJSON,
			Replicas:                    ConvertReplicas,
			InputFiles:                  GlobalFiles,
			OutFile:                     ConvertOut,
			Provider:                    GlobalProvider,
			CreateD:                     ConvertDeployment,
			CreateDS:                    ConvertDaemonSet,
			CreateRC:                    ConvertReplicationController,
			Build:                       ConvertBuild,
			BuildRepo:                   ConvertBuildRepo,
			BuildBranch:                 ConvertBuildBranch,
			PushImage:                   ConvertPushImage,
			CreateDeploymentConfig:      ConvertDeploymentConfig,
			EmptyVols:                   ConvertEmptyVols,
			Volumes:                     ConvertVolumes,
			InsecureRepository:          ConvertInsecureRepo,
			IsDeploymentFlag:            cmd.Flags().Lookup("deployment").Changed,
			IsDaemonSetFlag:             cmd.Flags().Lookup("daemon-set").Changed,
			IsReplicationControllerFlag: cmd.Flags().Lookup("replication-controller").Changed,
			Controller:                  strings.ToLower(ConvertController),
			IsReplicaSetFlag:            cmd.Flags().Lookup("replicas").Changed,
			IsDeploymentConfigFlag:      cmd.Flags().Lookup("deployment-config").Changed,
			YAMLIndent:                  ConvertYAMLIndent,
			WithKomposeAnnotation:       WithKomposeAnnotation,
			MultipleContainerMode:       MultipleContainerMode,
		}

		// Validate before doing anything else. Use "bundle" if passed in.
		app.ValidateFlags(GlobalBundle, args, cmd, &ConvertOpt)
		app.ValidateComposeFile(&ConvertOpt)
	},
	Run: func(cmd *cobra.Command, args []string) {

		app.Convert(ConvertOpt)
	},
}

func init() {
	// Automatically grab environment variables
	viper.AutomaticEnv()

	// Kubernetes only
	convertCmd.Flags().BoolVarP(&ConvertChart, "chart", "c", false, "Create a Helm chart for converted objects")
	convertCmd.Flags().BoolVar(&ConvertDaemonSet, "daemon-set", false, "Generate a Kubernetes daemonset object (deprecated, use --controller instead)")
	convertCmd.Flags().BoolVarP(&ConvertDeployment, "deployment", "d", false, "Generate a Kubernetes deployment object (deprecated, use --controller instead)")
	convertCmd.Flags().BoolVar(&ConvertReplicationController, "replication-controller", false, "Generate a Kubernetes replication controller object (deprecated, use --controller instead)")
	convertCmd.Flags().StringVar(&ConvertController, "controller", "", `Set the output controller ("deployment"|"daemonSet"|"replicationController")`)
	convertCmd.Flags().MarkDeprecated("daemon-set", "use --controller")
	convertCmd.Flags().MarkDeprecated("deployment", "use --controller")
	convertCmd.Flags().MarkDeprecated("replication-controller", "use --controller")
	convertCmd.Flags().MarkHidden("chart")
	convertCmd.Flags().MarkHidden("daemon-set")
	convertCmd.Flags().MarkHidden("replication-controller")
	convertCmd.Flags().MarkHidden("deployment")
	convertCmd.Flags().BoolVar(&MultipleContainerMode, "multiple-container-mode", false, "Create multiple containers grouped by 'kompose.service.group' label")

	// OpenShift only
	convertCmd.Flags().BoolVar(&ConvertDeploymentConfig, "deployment-config", true, "Generate an OpenShift deploymentconfig object")
	convertCmd.Flags().BoolVar(&ConvertInsecureRepo, "insecure-repository", false, "Use an insecure Docker repository for OpenShift ImageStream")
	convertCmd.Flags().StringVar(&ConvertBuildRepo, "build-repo", "", "Specify source repository for buildconfig (default remote origin)")
	convertCmd.Flags().StringVar(&ConvertBuildBranch, "build-branch", "", "Specify repository branch to use for buildconfig (default master)")
	convertCmd.Flags().MarkDeprecated("deployment-config", "use --controller")
	convertCmd.Flags().MarkHidden("deployment-config")
	convertCmd.Flags().MarkHidden("insecure-repository")
	convertCmd.Flags().MarkHidden("build-repo")
	convertCmd.Flags().MarkHidden("build-branch")

	// Standard between the two
	convertCmd.Flags().StringVar(&ConvertBuild, "build", "local", `Set the type of build ("local"|"build-config"(OpenShift only)|"none")`)
	convertCmd.Flags().BoolVar(&ConvertPushImage, "push-image", true, "If we should push the docker image we built")
	convertCmd.Flags().BoolVarP(&ConvertYaml, "yaml", "y", false, "Generate resource files into YAML format")
	convertCmd.Flags().MarkDeprecated("yaml", "YAML is the default format now.")
	convertCmd.Flags().MarkShorthandDeprecated("y", "YAML is the default format now.")
	convertCmd.Flags().BoolVarP(&ConvertJSON, "json", "j", false, "Generate resource files into JSON format")
	convertCmd.Flags().BoolVar(&ConvertStdout, "stdout", false, "Print converted objects to stdout")
	convertCmd.Flags().StringVarP(&ConvertOut, "out", "o", "", "Specify a file name or directory to save objects to (if path does not exist, a file will be created)")
	convertCmd.Flags().IntVar(&ConvertReplicas, "replicas", 1, "Specify the number of replicas in the generated resource spec")
	convertCmd.Flags().StringVar(&ConvertVolumes, "volumes", "persistentVolumeClaim", `Volumes to be generated ("persistentVolumeClaim"|"emptyDir"|"hostPath" | "configMap")`)

	convertCmd.Flags().BoolVar(&WithKomposeAnnotation, "with-kompose-annotation", true, "Add kompose annotations to generated resource")

	// Deprecated commands
	convertCmd.Flags().BoolVar(&ConvertEmptyVols, "emptyvols", false, "Use Empty Volumes. Do not generate PVCs")
	convertCmd.Flags().MarkDeprecated("emptyvols", "emptyvols has been marked as deprecated. Use --volumes empty")

	convertCmd.Flags().IntVar(&ConvertYAMLIndent, "indent", 2, "Spaces length to indent generated yaml files")

	// In order to 'separate' both OpenShift and Kubernetes only flags. A custom help page is created
	customHelp := `Usage:{{if .Runnable}}
  {{if .HasAvailableFlags}}{{appendIfNotPresent .UseLine "[flags]"}}{{else}}{{.UseLine}}{{end}}{{end}}{{if .HasAvailableSubCommands}}
  {{ .CommandPath}} [command]{{end}}{{if gt .Aliases 0}}

Aliases:
  {{.NameAndAliases}}
{{end}}{{if .HasExample}}

Examples:
{{ .Example }}{{end}}{{ if .HasAvailableSubCommands}}
Available Commands:{{range .Commands}}{{if .IsAvailableCommand}}
  {{rpad .Name .NamePadding }} {{.Short}}{{end}}{{end}}{{end}}{{ if .HasAvailableLocalFlags}}

Kubernetes Flags:
      --daemon-set               Generate a Kubernetes daemonset object (deprecated, use --controller instead)
  -d, --deployment               Generate a Kubernetes deployment object (deprecated, use --controller instead)
  -c, --chart                    Create a Helm chart for converted objects
      --replication-controller   Generate a Kubernetes replication controller object (deprecated, use --controller instead)

OpenShift Flags:
      --build-branch             Specify repository branch to use for buildconfig (default is current branch name)
      --build-repo               Specify source repository for buildconfig (default is current branch's remote url)
      --deployment-config        Generate an OpenShift deployment config object
      --insecure-repository      Specify to use insecure docker repository while generating Openshift image stream object

Flags:
{{.LocalFlags.FlagUsages | trimRightSpace}}{{end}}{{ if .HasAvailableInheritedFlags}}

Global Flags:
{{.InheritedFlags.FlagUsages | trimRightSpace}}{{end}}{{if .HasHelpSubCommands}}

Additional help topics:{{range .Commands}}{{if .IsHelpCommand}}
  {{rpad .CommandPath .CommandPathPadding}} {{.Short}}{{end}}{{end}}{{end}}{{ if .HasAvailableSubCommands }}
Use "{{.CommandPath}} [command] --help" for more information about a command.{{end}}
`
	// Set the help template + add the command to root
	convertCmd.SetUsageTemplate(customHelp)

	RootCmd.AddCommand(convertCmd)
}
<|endoftext|>


<commit_before>// Copyright 2017 Volker Dobler.  All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package main

import (
	"flag"
	"fmt"
	"log"
	"net/http"
	"os"

	"github.com/vdobler/ht/ht"
	"github.com/vdobler/ht/mock"
	"github.com/vdobler/ht/suite"
)

var cmdMock = &Command{
	RunArgs:     runMock,
	Usage:       "mock <mock>...",
	Description: "mock server",
	Flag:        flag.NewFlagSet("stat", flag.ContinueOnError),
	Help: `
Mock starts a HTTP server providing the given mocks.
`,
}

func init() {
	addVarsFlags(cmdMock.Flag)
	/*
		cmdMock.Flag.StringVar(&output, "output", "throughput.csv",
			"save results to `name`")
		cmdMock.Flag.BoolVar(&logplot, "log", true,
			"show logarithmic scale on plot")
		cmdMock.Flag.IntVar(&plotwidth, "plotwidth", 120,
			"draw plot `num` chars wide")
		cmdMock.Flag.DurationVar(&rampDuration, "ramp", 5*time.Second,
			"ramp duration to ignore while computing average QPS")
	*/
}

func runMock(cmd *Command, args []string) {
	// Sanity check.
	if len(args) == 0 {
		fmt.Fprintln(os.Stderr, "No mock given")
		fmt.Fprintf(os.Stderr, "Usage: %s\n", cmd.Usage)
		os.Exit(9)
	}

	monitor := make(chan *ht.Test)

	mocks := []*mock.Mock{}
	for _, arg := range args {
		m, err := suite.LoadMock(arg, nil, variablesFlag, false)
		if err != nil {
			fmt.Fprintln(os.Stderr, err.Error())
			os.Exit(8)
		}
		m.Monitor = monitor
		mocks = append(mocks, m)
	}

	logger := log.New(os.Stdout, "", 0)
	nfh := func(w http.ResponseWriter, r *http.Request) {
		fmt.Printf("Mock not found: 404 for %s %s\n", r.Method, r.URL)
		fmt.Println("%s", "===========================================================")
		http.Error(w, "Not found", 404)
	}
	_, err := mock.Serve(mocks, http.HandlerFunc(nfh), logger)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Problems staring server: %s\n", err)
		os.Exit(9)
	}

	for {
		report := <-monitor
		fmt.Println(mock.PrintReport(report))
	}
}
<commit_msg>cmd/ht: use new mock package<commit_after>// Copyright 2017 Volker Dobler.  All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package main

import (
	"flag"
	"fmt"
	"log"
	"net/http"
	"os"

	"github.com/vdobler/ht/ht"
	"github.com/vdobler/ht/mock"
	"github.com/vdobler/ht/suite"
)

var cmdMock = &Command{
	RunArgs:     runMock,
	Usage:       "mock <mock>...",
	Description: "mock server",
	Flag:        flag.NewFlagSet("stat", flag.ContinueOnError),
	Help: `
Mock starts a HTTP server providing the given mocks.
`,
}

var (
	certFile string
	keyFile  string
)

func init() {
	addVarsFlags(cmdMock.Flag)
	cmdMock.Flag.StringVar(&certFile, "cert", "",
		"load server certificate for https mocks from `file`")
	cmdMock.Flag.StringVar(&keyFile, "key", "",
		"load private key for https mocks from `file`")
}

func runMock(cmd *Command, args []string) {
	// Sanity check.
	if len(args) == 0 {
		fmt.Fprintln(os.Stderr, "No mock given")
		fmt.Fprintf(os.Stderr, "Usage: %s\n", cmd.Usage)
		os.Exit(9)
	}

	monitor := make(chan *ht.Test)

	mocks := []*mock.Mock{}
	for _, arg := range args {
		m, err := suite.LoadMock(arg, nil, variablesFlag, false)
		if err != nil {
			fmt.Fprintln(os.Stderr, err.Error())
			os.Exit(8)
		}
		m.Monitor = monitor
		mocks = append(mocks, m)
	}

	logger := log.New(os.Stdout, "", 0)
	nfh := func(w http.ResponseWriter, r *http.Request) {
		fmt.Printf("Mock not found: 404 for %s %s\n", r.Method, r.URL)
		fmt.Println("%s", "===========================================================")
		http.Error(w, "Not found", 404)
	}
	_, err := mock.Serve(mocks, http.HandlerFunc(nfh), logger, certFile, keyFile)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Problems staring server: %s\n", err)
		os.Exit(9)
	}

	for {
		report := <-monitor
		fmt.Println(mock.PrintReport(report))
	}
}
<|endoftext|>


<commit_before>package main

import (
	"flag"
	"fmt"
	"os"

	"github.com/simeji/jid"
)

const VERSION = "0.7.5"

func main() {
	content := os.Stdin

	var qm bool
	var help bool
	var version bool
	var mono bool
	var pretty bool
	qs := "."

	flag.BoolVar(&qm, "q", false, "Output query mode")
	flag.BoolVar(&help, "h", false, "print a help")
	flag.BoolVar(&help, "help", false, "print a help")
	flag.BoolVar(&version, "version", false, "print the version and exit")
	flag.BoolVar(&mono, "M", false, "monochrome output mode")
	flag.BoolVar(&pretty, "p", false, "pretty print json result")
	flag.Parse()

	if help {
		flag.Usage()
		fmt.Println(getHelpString())
		os.Exit(0)
	}

	if version {
		fmt.Println(fmt.Sprintf("jid version v%s", VERSION))
		os.Exit(0)
	}
	args := flag.Args()
	if len(args) > 0 {
		qs = args[0]
	}

	ea := &jid.EngineAttribute{
		DefaultQuery: qs,
		Monochrome:   mono,
		PrettyResult: pretty,
	}

	e, err := jid.NewEngine(content, ea)

	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	os.Exit(run(e, qm))
}

func run(e jid.EngineInterface, qm bool) int {

	result := e.Run()
	if result.GetError() != nil {
		return 2
	}
	if qm {
		fmt.Printf("%s", result.GetQueryString())
	} else {
		fmt.Printf("%s", result.GetContent())
	}
	return 0
}

func getHelpString() string {
	return `

============ Load JSON from a file ==============

$ jid < file.json

============ With a JSON filter mode =============

TAB / CTRL-I
  Show available items and choice them

CTRL-W
  Delete from the cursor to the start of the word

CTRL-U
  Delete whole query

CTRL-F / Right Arrow
  Move cursor a character to the right

CTRL-B / Left Arrow
  Move cursor a character to the left

CTRL-A
  To the first character of the 'Filter'

CTRL-E
  To the end of the 'Filter'

CTRL-J
  Scroll json buffer 1 line downwards

CTRL-K
  Scroll json buffer 1 line upwards

CTRL-G
  Scroll json buffer to bottom

CTRL-T
  Scroll json buffer to top

CTRL-N
  Scroll json buffer 'Page Down'

CTRL-P
  Scroll json buffer 'Page Up'

CTRL-L
  Change view mode whole json or keys (only object)

ESC
  Hide a candidate box

`
}
<commit_msg>fix version<commit_after>package main

import (
	"flag"
	"fmt"
	"os"

	"github.com/simeji/jid"
)

const VERSION = "0.7.6"

func main() {
	content := os.Stdin

	var qm bool
	var help bool
	var version bool
	var mono bool
	var pretty bool
	qs := "."

	flag.BoolVar(&qm, "q", false, "Output query mode")
	flag.BoolVar(&help, "h", false, "print a help")
	flag.BoolVar(&help, "help", false, "print a help")
	flag.BoolVar(&version, "version", false, "print the version and exit")
	flag.BoolVar(&mono, "M", false, "monochrome output mode")
	flag.BoolVar(&pretty, "p", false, "pretty print json result")
	flag.Parse()

	if help {
		flag.Usage()
		fmt.Println(getHelpString())
		os.Exit(0)
	}

	if version {
		fmt.Println(fmt.Sprintf("jid version v%s", VERSION))
		os.Exit(0)
	}
	args := flag.Args()
	if len(args) > 0 {
		qs = args[0]
	}

	ea := &jid.EngineAttribute{
		DefaultQuery: qs,
		Monochrome:   mono,
		PrettyResult: pretty,
	}

	e, err := jid.NewEngine(content, ea)

	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	os.Exit(run(e, qm))
}

func run(e jid.EngineInterface, qm bool) int {

	result := e.Run()
	if result.GetError() != nil {
		return 2
	}
	if qm {
		fmt.Printf("%s", result.GetQueryString())
	} else {
		fmt.Printf("%s", result.GetContent())
	}
	return 0
}

func getHelpString() string {
	return `

============ Load JSON from a file ==============

$ jid < file.json

============ With a JSON filter mode =============

TAB / CTRL-I
  Show available items and choice them

CTRL-W
  Delete from the cursor to the start of the word

CTRL-U
  Delete whole query

CTRL-F / Right Arrow
  Move cursor a character to the right

CTRL-B / Left Arrow
  Move cursor a character to the left

CTRL-A
  To the first character of the 'Filter'

CTRL-E
  To the end of the 'Filter'

CTRL-J
  Scroll json buffer 1 line downwards

CTRL-K
  Scroll json buffer 1 line upwards

CTRL-G
  Scroll json buffer to bottom

CTRL-T
  Scroll json buffer to top

CTRL-N
  Scroll json buffer 'Page Down'

CTRL-P
  Scroll json buffer 'Page Up'

CTRL-L
  Change view mode whole json or keys (only object)

ESC
  Hide a candidate box

`
}
<|endoftext|>


<commit_before>package cmd

import (
	"context"
	"fmt"
	"path/filepath"
	"strings"

	"github.com/jessevdk/go-flags"
	"github.com/pkg/errors"

	"github.com/mitchellh/cli"
	"github.com/nerdalize/nerd/pkg/transfer"
	"github.com/nerdalize/nerd/svc"
)

//JobRun command
type JobRun struct {
	KubeOpts
	TransferOpts
	Name    string   `long:"name" short:"n" description:"assign a name to the job"`
	Env     []string `long:"env" short:"e" description:"environment variables to use"`
	Inputs  []string `long:"input" description:"specify one or more inputs that will be downloaded for the job"`
	Outputs []string `long:"output" description:"specify one or more output folders that will be uploaded as datasets after the job is finished"`

	*command
}

//JobRunFactory creates the command
func JobRunFactory(ui cli.Ui) cli.CommandFactory {
	cmd := &JobRun{}
	cmd.command = createCommand(ui, cmd.Execute, cmd.Description, cmd.Usage, cmd, flags.PassAfterNonOption)
	return func() (cli.Command, error) {
		return cmd, nil
	}
}

//isPathAbsUnix checks if a path is absolute on a Unix system.
//Derived from https://github.com/golang/go/blob/1106512db54fc2736c7a9a67dd553fc9e1fca742/src/path/filepath/path_unix.go#L12
func isPathAbsUnix(path string) bool {
	return strings.HasPrefix(path, "/")
}

//Execute runs the command
func (cmd *JobRun) Execute(args []string) (err error) {
	if len(args) < 1 {
		return errShowUsage(MessageNotEnoughArguments)
	}

	kopts := cmd.KubeOpts
	deps, err := NewDeps(cmd.Logger(), kopts)
	if err != nil {
		return renderConfigError(err, "failed to configure")
	}

	ctx := context.Background()
	ctx, cancel := context.WithTimeout(ctx, cmd.Timeout)
	defer cancel()

	jargs := []string{}
	if len(args) > 1 {
		jargs = args[1:]
	}

	jenv := map[string]string{}
	for _, l := range cmd.Env {
		split := strings.SplitN(l, "=", 2)
		if len(split) < 2 {
			return fmt.Errorf("invalid environment variable format, expected 'FOO=bar' format, got: %v", l)
		}
		jenv[split[0]] = split[1]
	}

	kube := svc.NewKube(deps)

	//start with input volumes
	//@TODO move this logic to a separate package and test is
	var inputDataset string
	vols := map[string]*svc.JobVolume{}
	for _, input := range cmd.Inputs {
		parts := strings.Split(input, ":")
		if len(parts) != 2 {
			return fmt.Errorf("invalid input specified, expected '<DIR|DATASET_ID>:<JOB_DIR>' format, got: %s", input)
		}

		if !isPathAbsUnix(parts[1]) {
			return fmt.Errorf("the job directory for the input dataset must be provided as an absolute path")
		}

		//detect if the users tries to refer to a path to upload but it is not absolute
		if strings.Contains(parts[0], string(filepath.Separator)) && !filepath.IsAbs(parts[0]) {
			return fmt.Errorf("when providing a local directory as input it must be given as an absolute path")
		}

		//if the input spec has an absolute path as its first part, try to upload it for the user
		// var bucket string
		// var key string
		if filepath.IsAbs(parts[0]) {

			//the user has provided a path as its input
			var trans transfer.Transfer
			trans, err = cmd.TransferOpts.Transfer()
			if err != nil {
				return errors.Wrap(err, "failed configure transfer")
			}

			// var ref *transfer.Ref
			_, inputDataset, err = uploadToDataset(ctx, trans, cmd.AWSS3Bucket, kube, parts[0], "")
			if err != nil {
				return err
			}

			cmd.out.Infof("Uploaded input dataset: '%s'", inputDataset)
		} else {

			//the user (probably) has provided a dataset id as input
			var out *svc.GetDatasetOutput
			out, err = kube.GetDataset(ctx, &svc.GetDatasetInput{Name: parts[0]})
			if err != nil {
				return errors.Wrapf(err, "failed to get dataset '%s' ", parts[0])
			}

			if out.Bucket == "" || out.Key == "" {
				return errors.Errorf("the dataset '%s' cannot be used as input it has no key and/or bucket configured", parts[0])
			}

			inputDataset = out.Name
		}

		vols[parts[1]] = &svc.JobVolume{
			MountPath:    parts[1],
			InputDataset: inputDataset,
		}
	}

	var outputDataset string
	for _, output := range cmd.Outputs {
		parts := strings.Split(output, ":")
		if len(parts) < 1 || len(parts) > 2 {
			return fmt.Errorf("invalid output specified, expected '<JOB_DIR>:[DATASET_NAME]' format, got: %s", output)
		}

		if !isPathAbsUnix(parts[0]) {
			return fmt.Errorf("the job directory for the output dataset must be provided as an absolute path")
		}

		vol, ok := vols[parts[0]]
		if !ok {
			vol = &svc.JobVolume{MountPath: parts[0]}
			vols[parts[0]] = vol
		}

		if len(parts) == 2 {
			var out *svc.GetDatasetOutput
			out, err = kube.GetDataset(ctx, &svc.GetDatasetInput{Name: parts[1]})
			if err != nil {
				return errors.Wrapf(err, "failed to get dataset '%s' ", parts[1]) //@TODO do we want to hint the user that he might meant to specify a releative directory(?)
			}

			if out.Bucket == "" || out.Key == "" {
				return errors.Errorf("the dataset '%s' cannot be used as output as it has no key and/or bucket configured", parts[0])
			}

			outputDataset = out.Name
		} else {
			var trans transfer.Transfer
			trans, err = cmd.TransferOpts.Transfer()
			if err != nil {
				return errors.Wrap(err, "failed configure transfer")
			}

			_, outputDataset, err = uploadToDataset(ctx, trans, cmd.AWSS3Bucket, kube, "", "")
			if err != nil {
				return err
			}

			cmd.out.Infof("Setup empty output dataset: '%s'", outputDataset)
		}

		vol.OutputDataset = outputDataset
	}

	in := &svc.RunJobInput{
		Image: args[0],
		Name:  cmd.Name,
		Env:   jenv,
		Args:  jargs,
	}

	for _, vol := range vols {
		in.Volumes = append(in.Volumes, *vol)
	}

	out, err := kube.RunJob(ctx, in)
	if err != nil {
		return renderServiceError(err, "failed to run job")
	}

	err = updateDataset(ctx, inputDataset, outputDataset, out.Name, kube)
	if err != nil {
		return err
	}

	cmd.out.Infof("Submitted job: '%s'", out.Name)
	cmd.out.Infof("To see whats happening, use: 'nerd job list'")
	return nil
}

func updateDataset(ctx context.Context, inputDataset, outputDataset, job string, kube *svc.Kube) error {
	_, err := kube.UpdateDataset(ctx, &svc.UpdateDatasetInput{Name: inputDataset, InputFor: job})
	if err != nil {
		return err
	}
	_, err = kube.UpdateDataset(ctx, &svc.UpdateDatasetInput{Name: outputDataset, OutputFrom: job})
	if err != nil {
		return err
	}
	return nil
}

// Description returns long-form help text
func (cmd *JobRun) Description() string { return cmd.Synopsis() }

// Synopsis returns a one-line
func (cmd *JobRun) Synopsis() string { return "Runs a job on your compute cluster" }

// Usage shows usage
func (cmd *JobRun) Usage() string { return "nerd job run [OPTIONS] IMAGE [COMMAND] [ARG...]" }
<commit_msg>Modify job run to accept Windows paths with colons<commit_after>package cmd

import (
	"context"
	"fmt"
	"path/filepath"
	"strings"

	"github.com/jessevdk/go-flags"
	"github.com/pkg/errors"

	"github.com/mitchellh/cli"
	"github.com/nerdalize/nerd/pkg/transfer"
	"github.com/nerdalize/nerd/svc"
)

//JobRun command
type JobRun struct {
	KubeOpts
	TransferOpts
	Name    string   `long:"name" short:"n" description:"assign a name to the job"`
	Env     []string `long:"env" short:"e" description:"environment variables to use"`
	Inputs  []string `long:"input" description:"specify one or more inputs that will be downloaded for the job"`
	Outputs []string `long:"output" description:"specify one or more output folders that will be uploaded as datasets after the job is finished"`

	*command
}

//JobRunFactory creates the command
func JobRunFactory(ui cli.Ui) cli.CommandFactory {
	cmd := &JobRun{}
	cmd.command = createCommand(ui, cmd.Execute, cmd.Description, cmd.Usage, cmd, flags.PassAfterNonOption)
	return func() (cli.Command, error) {
		return cmd, nil
	}
}

//isPathAbsUnix checks if a path is absolute on a Unix system.
//Derived from https://github.com/golang/go/blob/1106512db54fc2736c7a9a67dd553fc9e1fca742/src/path/filepath/path_unix.go#L12
func isPathAbsUnix(path string) bool {
	return strings.HasPrefix(path, "/")
}

//Execute runs the command
func (cmd *JobRun) Execute(args []string) (err error) {
	if len(args) < 1 {
		return errShowUsage(MessageNotEnoughArguments)
	}

	kopts := cmd.KubeOpts
	deps, err := NewDeps(cmd.Logger(), kopts)
	if err != nil {
		return renderConfigError(err, "failed to configure")
	}

	ctx := context.Background()
	ctx, cancel := context.WithTimeout(ctx, cmd.Timeout)
	defer cancel()

	jargs := []string{}
	if len(args) > 1 {
		jargs = args[1:]
	}

	jenv := map[string]string{}
	for _, l := range cmd.Env {
		split := strings.SplitN(l, "=", 2)
		if len(split) < 2 {
			return fmt.Errorf("invalid environment variable format, expected 'FOO=bar' format, got: %v", l)
		}
		jenv[split[0]] = split[1]
	}

	kube := svc.NewKube(deps)

	//start with input volumes
	//@TODO move this logic to a separate package and test is
	var inputDataset string
	vols := map[string]*svc.JobVolume{}
	for _, input := range cmd.Inputs {
		parts := strings.Split(input, ":")
		if len(parts) < 1 {
			return fmt.Errorf("invalid input specified, expected '<DIR|DATASET_ID>:<JOB_DIR>' format, got: %s", input)
		}

		//Handle Windows paths where DIR may contain colons
		//e.g. C:/foo/bar:/input will be parsed into []string{"C", "/foo/bar", "/input"}
		//and should be turned into []string{"C:/foo/bar", "/input"}
		//We assume that POSIX paths will never have colons
		parts = []string{strings.Join(parts[:len(parts)-1], ":"), parts[len(parts)-1]}

		if !isPathAbsUnix(parts[1]) {
			return fmt.Errorf("the job directory for the input dataset must be provided as an absolute path")
		}

		//detect if the users tries to refer to a path to upload but it is not absolute
		if strings.Contains(parts[0], string(filepath.Separator)) && !filepath.IsAbs(parts[0]) {
			return fmt.Errorf("when providing a local directory as input it must be given as an absolute path")
		}

		//if the input spec has an absolute path as its first part, try to upload it for the user
		// var bucket string
		// var key string
		if filepath.IsAbs(parts[0]) {

			//the user has provided a path as its input
			var trans transfer.Transfer
			trans, err = cmd.TransferOpts.Transfer()
			if err != nil {
				return errors.Wrap(err, "failed configure transfer")
			}

			// var ref *transfer.Ref
			_, inputDataset, err = uploadToDataset(ctx, trans, cmd.AWSS3Bucket, kube, parts[0], "")
			if err != nil {
				return err
			}

			cmd.out.Infof("Uploaded input dataset: '%s'", inputDataset)
		} else {

			//the user (probably) has provided a dataset id as input
			var out *svc.GetDatasetOutput
			out, err = kube.GetDataset(ctx, &svc.GetDatasetInput{Name: parts[0]})
			if err != nil {
				return errors.Wrapf(err, "failed to get dataset '%s' ", parts[0])
			}

			if out.Bucket == "" || out.Key == "" {
				return errors.Errorf("the dataset '%s' cannot be used as input it has no key and/or bucket configured", parts[0])
			}

			inputDataset = out.Name
		}

		vols[parts[1]] = &svc.JobVolume{
			MountPath:    parts[1],
			InputDataset: inputDataset,
		}
	}

	var outputDataset string
	for _, output := range cmd.Outputs {
		parts := strings.Split(output, ":")
		if len(parts) < 1 || len(parts) > 2 {
			return fmt.Errorf("invalid output specified, expected '<JOB_DIR>:[DATASET_NAME]' format, got: %s", output)
		}

		if !isPathAbsUnix(parts[0]) {
			return fmt.Errorf("the job directory for the output dataset must be provided as an absolute path")
		}

		vol, ok := vols[parts[0]]
		if !ok {
			vol = &svc.JobVolume{MountPath: parts[0]}
			vols[parts[0]] = vol
		}

		if len(parts) == 2 {
			var out *svc.GetDatasetOutput
			out, err = kube.GetDataset(ctx, &svc.GetDatasetInput{Name: parts[1]})
			if err != nil {
				return errors.Wrapf(err, "failed to get dataset '%s' ", parts[1]) //@TODO do we want to hint the user that he might meant to specify a releative directory(?)
			}

			if out.Bucket == "" || out.Key == "" {
				return errors.Errorf("the dataset '%s' cannot be used as output as it has no key and/or bucket configured", parts[0])
			}

			outputDataset = out.Name
		} else {
			var trans transfer.Transfer
			trans, err = cmd.TransferOpts.Transfer()
			if err != nil {
				return errors.Wrap(err, "failed configure transfer")
			}

			_, outputDataset, err = uploadToDataset(ctx, trans, cmd.AWSS3Bucket, kube, "", "")
			if err != nil {
				return err
			}

			cmd.out.Infof("Setup empty output dataset: '%s'", outputDataset)
		}

		vol.OutputDataset = outputDataset
	}

	in := &svc.RunJobInput{
		Image: args[0],
		Name:  cmd.Name,
		Env:   jenv,
		Args:  jargs,
	}

	for _, vol := range vols {
		in.Volumes = append(in.Volumes, *vol)
	}

	out, err := kube.RunJob(ctx, in)
	if err != nil {
		return renderServiceError(err, "failed to run job")
	}

	err = updateDataset(ctx, inputDataset, outputDataset, out.Name, kube)
	if err != nil {
		return err
	}

	cmd.out.Infof("Submitted job: '%s'", out.Name)
	cmd.out.Infof("To see whats happening, use: 'nerd job list'")
	return nil
}

func updateDataset(ctx context.Context, inputDataset, outputDataset, job string, kube *svc.Kube) error {
	_, err := kube.UpdateDataset(ctx, &svc.UpdateDatasetInput{Name: inputDataset, InputFor: job})
	if err != nil {
		return err
	}
	_, err = kube.UpdateDataset(ctx, &svc.UpdateDatasetInput{Name: outputDataset, OutputFrom: job})
	if err != nil {
		return err
	}
	return nil
}

// Description returns long-form help text
func (cmd *JobRun) Description() string { return cmd.Synopsis() }

// Synopsis returns a one-line
func (cmd *JobRun) Synopsis() string { return "Runs a job on your compute cluster" }

// Usage shows usage
func (cmd *JobRun) Usage() string { return "nerd job run [OPTIONS] IMAGE [COMMAND] [ARG...]" }
<|endoftext|>


<commit_before>package main

// jwt either encodes or decodes stdin, writing to stdout using provided key
// data.
//
//
// Example:
//		# decode and verify a token
//		echo "" |jwt -dec -k rsa.pem
//		echo ""t

import (
	"bytes"
	"crypto/ecdsa"
	"crypto/elliptic"
	"crypto/rsa"
	"encoding/json"
	"errors"
	"flag"
	"fmt"
	"io/ioutil"
	"os"
	"strconv"
	"strings"

	"github.com/knq/jwt"
	"github.com/knq/pemutil"
)

var (
	flagEnc = flag.Bool("enc", false, "encode token from json data sent via stdin, or via name=value pairs passed on the command line.")
	flagDec = flag.Bool("dec", false, "decode token read from stdin")
	flagKey = flag.String("k", "", "path to PEM-encoded file containing key data")
	flagAlg = flag.String("alg", "", "use specified algorithm")
)

func main() {
	var err error

	// parse parameters
	flag.Parse()

	// make sure k parameter is specified
	if *flagKey == "" {
		fmt.Fprintln(os.Stderr, "error: must supply a key")
		os.Exit(1)
	}

	// read key data
	pem := pemutil.Store{}
	err = pemutil.PEM{*flagKey}.Load(pem)
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %s\n", err)
		os.Exit(1)
	}

	// set key
	var alg jwt.Algorithm

	// get alg from key
	if *flagAlg == "" {
		alg, err = getAlgFromKeyData(pem)
		if err != nil {
			fmt.Fprintf(os.Stderr, "error: %v\n", err)
			os.Exit(1)
		}
	} else {
		// attempt to decode alg
		err = json.Unmarshal([]byte(`"`+*flagAlg+`"`), &alg)
		if err != nil {
			fmt.Fprintf(os.Stderr, "error: %v\n", err)
			os.Exit(1)
		}
	}

	// create signer
	signer := alg.New(pem)

	// inspect remaining args
	args := flag.Args()
	if len(args) > 0 && *flagDec {
		fmt.Fprintln(os.Stderr, "error: unknown args passed for -dec")
		os.Exit(1)
	}

	var in []byte

	// if there are command line args and enc, then build js from them
	if len(args) > 0 && *flagEnc {
		in, err = buildEncArgs(args)
	} else {
		in, err = ioutil.ReadAll(os.Stdin)
	}

	// check errors
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}

	// encode, decode, or error
	var out []byte
	switch {
	case *flagDec:
		out, err = doDec(signer, in)

	case *flagEnc:
		out, err = doEnc(signer, in)

	default:
		err = errors.New("please specify -enc or -dec")
	}

	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}

	os.Stdout.Write(out)
}

// getSuitableAlgFromCurve inspects the key length in curve, and determines the
// corresponding jwt.Algorithm.
func getSuitableAlgFromCurve(curve elliptic.Curve) (jwt.Algorithm, error) {
	curveBitSize := curve.Params().BitSize

	// compute curve key len
	keyLen := curveBitSize / 8
	if curveBitSize%8 > 0 {
		keyLen++
	}

	// determine alg
	var alg jwt.Algorithm
	switch 2 * keyLen {
	case 64:
		alg = jwt.ES256
	case 96:
		alg = jwt.ES384
	case 132:
		alg = jwt.ES512

	default:
		return jwt.NONE, fmt.Errorf("invalid key length %d", keyLen)
	}

	return alg, nil
}

// getAlgFromKeyData determines the best jwt.Algorithm suitable based on the
// set of given crypto primitives in pem.
func getAlgFromKeyData(pem pemutil.Store) (jwt.Algorithm, error) {
	for _, v := range pem {
		// loop over crypto primitives in pemstore, and do type assertion. if
		// ecdsa.{PublicKey,PrivateKey} found, then use corresponding ESXXX as
		// algo. if rsa, then use DefaultRSAAlgorithm. if []byte, then use
		// DefaultHMACAlgorithm.
		switch k := v.(type) {
		case []byte:
			return jwt.HS512, nil

		case *ecdsa.PrivateKey:
			return getSuitableAlgFromCurve(k.Curve)

		case *ecdsa.PublicKey:
			return getSuitableAlgFromCurve(k.Curve)

		case *rsa.PrivateKey:
			return jwt.PS512, nil

		case *rsa.PublicKey:
			return jwt.PS512, nil
		}
	}

	return jwt.NONE, errors.New("cannot determine key type")
}

// buildEncArgs builds and encodes passed argument strings in the form of
// name=val as a json object.
func buildEncArgs(args []string) ([]byte, error) {
	m := make(map[string]interface{})

	// loop over args, splitting on '=', and attempt parsing of value
	for _, arg := range args {
		a := strings.SplitN(arg, "=", 2)

		var val interface{}
		if len(a) == 1 { // assume bool, set as true
			val = true
		} else if u, err := strconv.ParseUint(a[1], 10, 64); err == nil {
			val = u
		} else if i, err := strconv.ParseInt(a[1], 10, 64); err == nil {
			val = i
		} else if f, err := strconv.ParseFloat(a[1], 64); err == nil {
			val = f
		} else if b, err := strconv.ParseBool(a[1]); err == nil {
			val = b
		} else if s, err := strconv.Unquote(a[1]); err == nil {
			val = s
		} else { // treat as string
			val = a[1]
		}
		m[a[0]] = val
	}

	return json.Marshal(m)
}

// UnstructuredToken is a jwt compatible token for encoding/decoding unknown
// jwt payloads.
type UnstructuredToken struct {
	Header    map[string]interface{} `json:"header" jwt:"header"`
	Payload   map[string]interface{} `json:"payload" jwt:"payload"`
	Signature []byte                 `json:"signature" jwt:"signature"`
}

// doDec decodes in as a JWT.
func doDec(signer jwt.Signer, in []byte) ([]byte, error) {
	var err error

	// decode token
	ut := UnstructuredToken{}
	err = signer.Decode(bytes.TrimSpace(in), &ut)
	if err != nil {
		return nil, err
	}

	// pretty format output
	out, err := json.MarshalIndent(&ut, "", "  ")
	if err != nil {
		return nil, err
	}

	return out, nil
}

// doEnc encodes in as the payload in a JWT.
func doEnc(signer jwt.Signer, in []byte) ([]byte, error) {
	var err error

	// make sure its valid json first
	m := make(map[string]interface{})
	err = json.Unmarshal(in, &m)
	if err != nil {
		return nil, err
	}

	// encode claims
	out, err := signer.Encode(&m)
	if err != nil {
		return nil, err
	}

	return out, nil
}
<commit_msg>Minor code cleanup<commit_after>package main

// jwt either encodes or decodes stdin, writing to stdout using provided key
// data.
//
//
// Example:
//		# decode and verify a token
//		echo "" |jwt -dec -k rsa.pem
//		echo ""t

import (
	"bytes"
	"crypto/ecdsa"
	"crypto/elliptic"
	"crypto/rsa"
	"encoding/json"
	"errors"
	"flag"
	"fmt"
	"io/ioutil"
	"os"
	"strconv"
	"strings"

	"github.com/knq/jwt"
	"github.com/knq/pemutil"
)

var (
	flagEnc = flag.Bool("enc", false, "encode token from json data sent via stdin, or via name=value pairs passed on the command line.")
	flagDec = flag.Bool("dec", false, "decode token read from stdin")
	flagKey = flag.String("k", "", "path to PEM-encoded file containing key data")
	flagAlg = flag.String("alg", "", "use specified algorithm")
)

func main() {
	var err error

	// parse parameters
	flag.Parse()

	// make sure k parameter is specified
	if *flagKey == "" {
		fmt.Fprintln(os.Stderr, "error: must supply a key")
		os.Exit(1)
	}

	// read key data
	pem := pemutil.Store{}
	err = pemutil.PEM{*flagKey}.Load(pem)
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %s\n", err)
		os.Exit(1)
	}

	// set key
	var alg jwt.Algorithm

	// get alg from key
	if *flagAlg == "" {
		alg, err = getAlgFromKeyData(pem)
		if err != nil {
			fmt.Fprintf(os.Stderr, "error: %v\n", err)
			os.Exit(1)
		}
	} else {
		// attempt to decode alg
		err = json.Unmarshal([]byte(`"`+*flagAlg+`"`), &alg)
		if err != nil {
			fmt.Fprintf(os.Stderr, "error: %v\n", err)
			os.Exit(1)
		}
	}

	// create signer
	signer := alg.New(pem)

	// inspect remaining args
	args := flag.Args()
	if len(args) > 0 && *flagDec {
		fmt.Fprintln(os.Stderr, "error: unknown args passed for -dec")
		os.Exit(1)
	}

	var in []byte

	// if there are command line args and enc, then build js from them
	if len(args) > 0 && *flagEnc {
		in, err = buildEncArgs(args)
	} else {
		in, err = ioutil.ReadAll(os.Stdin)
	}

	// check errors
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}

	// encode, decode, or error
	var out []byte
	switch {
	case *flagDec:
		out, err = doDec(signer, in)

	case *flagEnc:
		out, err = doEnc(signer, in)

	default:
		err = errors.New("please specify -enc or -dec")
	}

	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}

	os.Stdout.Write(out)
}

// getSuitableAlgFromCurve inspects the key length in curve, and determines the
// corresponding jwt.Algorithm.
func getSuitableAlgFromCurve(curve elliptic.Curve) (jwt.Algorithm, error) {
	curveBitSize := curve.Params().BitSize

	// compute curve key len
	keyLen := curveBitSize / 8
	if curveBitSize%8 > 0 {
		keyLen++
	}

	// determine alg
	var alg jwt.Algorithm
	switch 2 * keyLen {
	case 64:
		alg = jwt.ES256
	case 96:
		alg = jwt.ES384
	case 132:
		alg = jwt.ES512

	default:
		return jwt.NONE, fmt.Errorf("invalid key length %d", keyLen)
	}

	return alg, nil
}

// getAlgFromKeyData determines the best jwt.Algorithm suitable based on the
// set of given crypto primitives in pem.
func getAlgFromKeyData(pem pemutil.Store) (jwt.Algorithm, error) {
	for _, v := range pem {
		// loop over crypto primitives in pemstore, and do type assertion. if
		// ecdsa.{PublicKey,PrivateKey} found, then use corresponding ESXXX as
		// algo. if rsa, then use DefaultRSAAlgorithm. if []byte, then use
		// DefaultHMACAlgorithm.
		switch k := v.(type) {
		case []byte:
			return jwt.HS512, nil

		case *ecdsa.PrivateKey:
			return getSuitableAlgFromCurve(k.Curve)

		case *ecdsa.PublicKey:
			return getSuitableAlgFromCurve(k.Curve)

		case *rsa.PrivateKey:
			return jwt.PS512, nil

		case *rsa.PublicKey:
			return jwt.PS512, nil
		}
	}

	return jwt.NONE, errors.New("cannot determine key type")
}

// buildEncArgs builds and encodes passed argument strings in the form of
// name=val as a json object.
func buildEncArgs(args []string) ([]byte, error) {
	m := make(map[string]interface{})

	// loop over args, splitting on '=', and attempt parsing of value
	for _, arg := range args {
		a := strings.SplitN(arg, "=", 2)
		var val interface{}

		// attempt to parse
		if len(a) == 1 { // assume bool, set as true
			val = true
		} else if u, err := strconv.ParseUint(a[1], 10, 64); err == nil {
			val = u
		} else if i, err := strconv.ParseInt(a[1], 10, 64); err == nil {
			val = i
		} else if f, err := strconv.ParseFloat(a[1], 64); err == nil {
			val = f
		} else if b, err := strconv.ParseBool(a[1]); err == nil {
			val = b
		} else if s, err := strconv.Unquote(a[1]); err == nil {
			val = s
		} else { // treat as string
			val = a[1]
		}

		m[a[0]] = val
	}

	return json.Marshal(m)
}

// UnstructuredToken is a jwt compatible token for encoding/decoding unknown
// jwt payloads.
type UnstructuredToken struct {
	Header    map[string]interface{} `json:"header" jwt:"header"`
	Payload   map[string]interface{} `json:"payload" jwt:"payload"`
	Signature []byte                 `json:"signature" jwt:"signature"`
}

// doDec decodes in as a JWT.
func doDec(signer jwt.Signer, in []byte) ([]byte, error) {
	var err error

	// decode token
	ut := UnstructuredToken{}
	err = signer.Decode(bytes.TrimSpace(in), &ut)
	if err != nil {
		return nil, err
	}

	// pretty format output
	out, err := json.MarshalIndent(&ut, "", "  ")
	if err != nil {
		return nil, err
	}

	return out, nil
}

// doEnc encodes in as the payload in a JWT.
func doEnc(signer jwt.Signer, in []byte) ([]byte, error) {
	var err error

	// make sure its valid json first
	m := make(map[string]interface{})
	err = json.Unmarshal(in, &m)
	if err != nil {
		return nil, err
	}

	// encode claims
	out, err := signer.Encode(&m)
	if err != nil {
		return nil, err
	}

	return out, nil
}
<|endoftext|>


<commit_before>package main

import (
	"bytes"
	"encoding/json"
	"flag"
	"fmt"
	"github.com/edmonds/golang-mtbl"
	"github.com/hdm/inetdata-parsers/utils"
	"github.com/peterbourgon/mergemap"
	"io/ioutil"
	"math"
	"net"
	"os"
	"runtime"
	"strings"
)

var key_only *bool
var val_only *bool
var prefix *string
var rev_prefix *string
var rev_key *bool
var no_quotes *bool
var as_json *bool
var version *bool
var domain *string
var cidr *string

func usage() {
	fmt.Println("Usage: " + os.Args[0] + " [options] <mtbl> ... <mtbl>")
	fmt.Println("")
	fmt.Println("Queries one or more MTBL databases")
	fmt.Println("")
	fmt.Println("Options:")
	flag.PrintDefaults()
}

func findPaths(args []string) []string {
	var paths []string
	for i := range args {
		path := args[i]
		info, e := os.Stat(path)
		if e != nil {
			fmt.Fprintf(os.Stderr, "Error: Path %s : %v\n", path, e)
			os.Exit(1)
		}

		if info.Mode().IsRegular() {
			paths = append(paths, path)
			continue
		}

		if info.Mode().IsDir() {
			if files, e := ioutil.ReadDir(path); e == nil {
				for _, f := range files {
					if f.Mode().IsRegular() {
						npath := path + string(os.PathSeparator) + f.Name()
						paths = append(paths, npath)
					}
				}
			}
		}
	}
	return paths
}

// Handles json formats like: {"k1": "val1"} and [ ['k1', 'v1'] ]
func mergeFunc(key []byte, val0 []byte, val1 []byte) (mergedVal []byte) {

	if bytes.Compare(val0, val1) == 0 {
		return val0
	}

	// Try to merge as a map[string]interface{}
	var v0, v1 map[string]interface{}

	if e := json.Unmarshal(val0, &v0); e == nil {
		// Looks like a map[string]interface{}

		// Try to unmarshal the second value the same way
		if e := json.Unmarshal(val1, &v1); e != nil {
			// Second value was not equivalent, return first value
			return val0
		}

		m := mergemap.Merge(v0, v1)
		d, e := json.Marshal(m)
		if e != nil {
			fmt.Fprintf(os.Stderr, "JSON merge error: %s -> %s + %s: %s\n", string(key), string(val0), string(val1), e.Error())
			return val0
		}

		return d
	}

	// Try to merge as a [][]string
	var a0, a1 [][]string

	if e := json.Unmarshal(val0, &a0); e == nil {
		// Looks like a [][]string

		// Try to unmarshal the second value the same way
		if e := json.Unmarshal(val1, &a1); e != nil {
			// Couldn't unmarshal the second value, return val0
			return val0
		}

		unique := map[string]bool{}
		m := [][]string{}

		for i := range a0 {
			unique[strings.Join(a0[i], "\x00")] = true
		}
		for i := range a1 {
			unique[strings.Join(a1[i], "\x00")] = true
		}
		for i := range unique {
			m = append(m, strings.SplitN(i, "\x00", 2))
		}

		d, e := json.Marshal(m)
		if e != nil {
			fmt.Fprintf(os.Stderr, "JSON merge error: %s -> %s + %s: %s\n", string(key), string(val0), string(val1), e.Error())
			return val0
		}

		return d
	}

	// Give up and return the first value
	return val0
}

func writeOutput(key_bytes []byte, val_bytes []byte) {

	key := string(key_bytes)
	val := string(val_bytes)

	if *rev_key {
		key = utils.ReverseKey(key)
	}

	if *as_json {
		o := make(map[string]interface{})
		v := make([][]string, 1)

		if de := json.Unmarshal([]byte(val), &v); de != nil {
			fmt.Fprintf(os.Stderr, "Could not unmarshal %s -> %s as json: %s\n", key, val, de)
			return
		}

		o["key"] = string(key)
		o["val"] = v

		b, je := json.Marshal(o)
		if je != nil {
			fmt.Fprintf(os.Stderr, "Could not marshal %s -> %s as json: %s\n", key, val, je)
			return
		}
		fmt.Println(string(b))

	} else if *key_only {
		fmt.Printf("%s\n", key)
	} else if *val_only {
		if *no_quotes {
			fmt.Printf("%s\n", val)
		} else {
			fmt.Printf("%q\n", val)
		}
	} else {
		if *no_quotes {
			fmt.Printf("%s\t%s\n", key, val)
		} else {
			fmt.Printf("%s\t%q\n", key, val)
		}
	}
}

func searchPrefix(m *mtbl.Merger, prefix string) {
	it := mtbl.IterPrefix(m, []byte(prefix))
	for {
		key_bytes, val_bytes, ok := it.Next()
		if !ok {
			break
		}
		writeOutput(key_bytes, val_bytes)
	}
}

func searchAll(m *mtbl.Merger) {
	it := mtbl.IterAll(m)
	for {
		key_bytes, val_bytes, ok := it.Next()
		if !ok {
			break
		}
		writeOutput(key_bytes, val_bytes)
	}
}

func searchDomain(m *mtbl.Merger, domain string) {
	rdomain := []byte(utils.ReverseKey(domain))

	// Domain searches always use reversed keys
	*rev_key = true

	// Exact match: "example.com"
	exact, found := mtbl.Get(m, rdomain)
	if found {
		writeOutput([]byte(rdomain), exact)
	}

	// Subdomain matches: ".example.com"
	dot_domain := append(rdomain, '.')
	it := mtbl.IterPrefix(m, dot_domain)
	for {
		key_bytes, val_bytes, ok := it.Next()
		if !ok {
			break
		}
		writeOutput(key_bytes, val_bytes)
	}
}

func searchPrefixIPv4(m *mtbl.Merger, prefix string) {
	it := mtbl.IterPrefix(m, []byte(prefix))
	for {
		key_bytes, val_bytes, ok := it.Next()
		if !ok {
			break
		}

		if utils.Match_IPv4.Match(key_bytes) {
			writeOutput(key_bytes, val_bytes)
		}
	}
}

func searchCIDR(m *mtbl.Merger, cidr string) {

	// Parse CIDR into base address + mask
	ip, net, err := net.ParseCIDR(cidr)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Invalid CIDR %s: %s", cidr, err.Error())
		return
	}

	// Verify IPv4 for now
	ip4 := net.IP.To4()
	if ip4 == nil {
		fmt.Fprintf(os.Stderr, "Invalid IPv4 CIDR %s\n", cidr)
		return
	}

	net_base, err := utils.IPv4_to_UInt(net.IP.String())
	if err != nil {
		fmt.Fprintf(os.Stderr, "Invalid IPv4 Address %s: %s", ip.String(), err.Error())
		return
	}

	mask_ones, mask_total := net.Mask.Size()

	// Does not work for IPv6 due to cast to uint32
	net_size := uint32(math.Pow(2, float64(mask_total-mask_ones)))

	cur_base := net_base
	end_base := net_base + net_size

	var ndots uint32 = 3
	var block_size uint32 = 256

	// Handle massive network blocks
	if mask_ones <= 8 {
		ndots = 1
		block_size = 256 * 256 * 256
	} else if mask_ones <= 16 {
		ndots = 2
		block_size = 256 * 256
	}

	// Iterate by block size
	for ; end_base-cur_base >= block_size; cur_base += block_size {
		ip_prefix := strings.Join(strings.SplitN(utils.UInt_to_IPv4(cur_base), ".", 4)[0:ndots], ".") + "."
		searchPrefixIPv4(m, ip_prefix)
	}

	if end_base-cur_base == 0 {
		return
	}

	// Handle any leftovers by looking up a full /24 and ignoring stuff outside our range
	ip_prefix := strings.Join(strings.SplitN(utils.UInt_to_IPv4(cur_base), ".", 4)[0:3], ".") + "."

	it := mtbl.IterPrefix(m, []byte(ip_prefix))
	for {
		key_bytes, val_bytes, ok := it.Next()
		if !ok {
			break
		}

		// Only print results that are valid IPV4 addresses within our CIDR range
		cur_val, _ := utils.IPv4_to_UInt(string(key_bytes))
		if cur_val >= cur_base && cur_val <= end_base {
			if utils.Match_IPv4.Match(key_bytes) {
				writeOutput(key_bytes, val_bytes)
			}
		}
	}
}

func main() {

	runtime.GOMAXPROCS(runtime.NumCPU())
	os.Setenv("LC_ALL", "C")

	flag.Usage = func() { usage() }

	key_only = flag.Bool("k", false, "Display key names only")
	val_only = flag.Bool("v", false, "Display values only")
	prefix = flag.String("p", "", "Only return keys with this prefix")
	rev_prefix = flag.String("r", "", "Only return keys with this prefix in reverse form")
	rev_key = flag.Bool("R", false, "Display matches with the key in reverse form")
	no_quotes = flag.Bool("n", false, "Print raw values, not quoted values")
	as_json = flag.Bool("j", false, "Print each record as a single line of JSON")
	version = flag.Bool("version", false, "Show the version and build timestamp")
	domain = flag.String("domain", "", "Search for all matches for a specified domain")
	cidr = flag.String("cidr", "", "Search for all matches for the specified CIDR")

	flag.Parse()

	if *version {
		utils.PrintVersion()
		os.Exit(0)
	}

	if len(flag.Args()) == 0 {
		usage()
		os.Exit(1)
	}

	if *key_only && *val_only {
		fmt.Fprintf(os.Stderr, "Error: Only one of -k or -v can be specified\n")
		usage()
		os.Exit(1)
	}

	if len(*prefix) > 0 && len(*rev_prefix) > 0 {
		fmt.Fprintf(os.Stderr, "Error: Only one of -p or -r can be specified\n")
		usage()
		os.Exit(1)
	}

	if len(*domain) > 0 && (len(*prefix) > 0 || len(*rev_prefix) > 0 || len(*cidr) > 0) {
		fmt.Fprintf(os.Stderr, "Error: Only one of -p, -r, -domain, or -cidr can be specified\n")
		usage()
		os.Exit(1)
	}

	if len(*cidr) > 0 && (len(*prefix) > 0 || len(*rev_prefix) > 0 || len(*domain) > 0) {
		fmt.Fprintf(os.Stderr, "Error: Only one of -p, -r, -domain, or -cidr can be specified\n")
		usage()
		os.Exit(1)
	}

	paths := findPaths(flag.Args())

	m := mtbl.MergerInit(&mtbl.MergerOptions{Merge: mergeFunc})
	defer m.Destroy()

	for i := range paths {
		path := paths[i]

		r, e := mtbl.ReaderInit(path, &mtbl.ReaderOptions{VerifyChecksums: true})
		defer r.Destroy()
		if e != nil {
			fmt.Fprintf(os.Stderr, "Error reading %s: %s\n", path, e)
			os.Exit(1)
		}
		m.AddSource(r)
	}

	if len(*domain) > 0 {
		searchDomain(m, *domain)
		return
	}

	if len(*cidr) > 0 {
		searchCIDR(m, *cidr)
		return
	}

	if len(*prefix) > 0 {
		searchPrefix(m, *prefix)
		return
	}

	if len(*rev_prefix) > 0 {
		p := utils.ReverseKey(*rev_prefix)
		searchPrefix(m, p)
		return
	}

	searchAll(m)
}
<commit_msg>Add missing newlines<commit_after>package main

import (
	"bytes"
	"encoding/json"
	"flag"
	"fmt"
	"github.com/edmonds/golang-mtbl"
	"github.com/hdm/inetdata-parsers/utils"
	"github.com/peterbourgon/mergemap"
	"io/ioutil"
	"math"
	"net"
	"os"
	"runtime"
	"strings"
)

var key_only *bool
var val_only *bool
var prefix *string
var rev_prefix *string
var rev_key *bool
var no_quotes *bool
var as_json *bool
var version *bool
var domain *string
var cidr *string

func usage() {
	fmt.Println("Usage: " + os.Args[0] + " [options] <mtbl> ... <mtbl>")
	fmt.Println("")
	fmt.Println("Queries one or more MTBL databases")
	fmt.Println("")
	fmt.Println("Options:")
	flag.PrintDefaults()
}

func findPaths(args []string) []string {
	var paths []string
	for i := range args {
		path := args[i]
		info, e := os.Stat(path)
		if e != nil {
			fmt.Fprintf(os.Stderr, "Error: Path %s : %v\n", path, e)
			os.Exit(1)
		}

		if info.Mode().IsRegular() {
			paths = append(paths, path)
			continue
		}

		if info.Mode().IsDir() {
			if files, e := ioutil.ReadDir(path); e == nil {
				for _, f := range files {
					if f.Mode().IsRegular() {
						npath := path + string(os.PathSeparator) + f.Name()
						paths = append(paths, npath)
					}
				}
			}
		}
	}
	return paths
}

// Handles json formats like: {"k1": "val1"} and [ ['k1', 'v1'] ]
func mergeFunc(key []byte, val0 []byte, val1 []byte) (mergedVal []byte) {

	if bytes.Compare(val0, val1) == 0 {
		return val0
	}

	// Try to merge as a map[string]interface{}
	var v0, v1 map[string]interface{}

	if e := json.Unmarshal(val0, &v0); e == nil {
		// Looks like a map[string]interface{}

		// Try to unmarshal the second value the same way
		if e := json.Unmarshal(val1, &v1); e != nil {
			// Second value was not equivalent, return first value
			return val0
		}

		m := mergemap.Merge(v0, v1)
		d, e := json.Marshal(m)
		if e != nil {
			fmt.Fprintf(os.Stderr, "JSON merge error: %s -> %s + %s: %s\n", string(key), string(val0), string(val1), e.Error())
			return val0
		}

		return d
	}

	// Try to merge as a [][]string
	var a0, a1 [][]string

	if e := json.Unmarshal(val0, &a0); e == nil {
		// Looks like a [][]string

		// Try to unmarshal the second value the same way
		if e := json.Unmarshal(val1, &a1); e != nil {
			// Couldn't unmarshal the second value, return val0
			return val0
		}

		unique := map[string]bool{}
		m := [][]string{}

		for i := range a0 {
			unique[strings.Join(a0[i], "\x00")] = true
		}
		for i := range a1 {
			unique[strings.Join(a1[i], "\x00")] = true
		}
		for i := range unique {
			m = append(m, strings.SplitN(i, "\x00", 2))
		}

		d, e := json.Marshal(m)
		if e != nil {
			fmt.Fprintf(os.Stderr, "JSON merge error: %s -> %s + %s: %s\n", string(key), string(val0), string(val1), e.Error())
			return val0
		}

		return d
	}

	// Give up and return the first value
	return val0
}

func writeOutput(key_bytes []byte, val_bytes []byte) {

	key := string(key_bytes)
	val := string(val_bytes)

	if *rev_key {
		key = utils.ReverseKey(key)
	}

	if *as_json {
		o := make(map[string]interface{})
		v := make([][]string, 1)

		if de := json.Unmarshal([]byte(val), &v); de != nil {
			fmt.Fprintf(os.Stderr, "Could not unmarshal %s -> %s as json: %s\n", key, val, de)
			return
		}

		o["key"] = string(key)
		o["val"] = v

		b, je := json.Marshal(o)
		if je != nil {
			fmt.Fprintf(os.Stderr, "Could not marshal %s -> %s as json: %s\n", key, val, je)
			return
		}
		fmt.Println(string(b))

	} else if *key_only {
		fmt.Printf("%s\n", key)
	} else if *val_only {
		if *no_quotes {
			fmt.Printf("%s\n", val)
		} else {
			fmt.Printf("%q\n", val)
		}
	} else {
		if *no_quotes {
			fmt.Printf("%s\t%s\n", key, val)
		} else {
			fmt.Printf("%s\t%q\n", key, val)
		}
	}
}

func searchPrefix(m *mtbl.Merger, prefix string) {
	it := mtbl.IterPrefix(m, []byte(prefix))
	for {
		key_bytes, val_bytes, ok := it.Next()
		if !ok {
			break
		}
		writeOutput(key_bytes, val_bytes)
	}
}

func searchAll(m *mtbl.Merger) {
	it := mtbl.IterAll(m)
	for {
		key_bytes, val_bytes, ok := it.Next()
		if !ok {
			break
		}
		writeOutput(key_bytes, val_bytes)
	}
}

func searchDomain(m *mtbl.Merger, domain string) {
	rdomain := []byte(utils.ReverseKey(domain))

	// Domain searches always use reversed keys
	*rev_key = true

	// Exact match: "example.com"
	exact, found := mtbl.Get(m, rdomain)
	if found {
		writeOutput([]byte(rdomain), exact)
	}

	// Subdomain matches: ".example.com"
	dot_domain := append(rdomain, '.')
	it := mtbl.IterPrefix(m, dot_domain)
	for {
		key_bytes, val_bytes, ok := it.Next()
		if !ok {
			break
		}
		writeOutput(key_bytes, val_bytes)
	}
}

func searchPrefixIPv4(m *mtbl.Merger, prefix string) {
	it := mtbl.IterPrefix(m, []byte(prefix))
	for {
		key_bytes, val_bytes, ok := it.Next()
		if !ok {
			break
		}

		if utils.Match_IPv4.Match(key_bytes) {
			writeOutput(key_bytes, val_bytes)
		}
	}
}

func searchCIDR(m *mtbl.Merger, cidr string) {

	// Parse CIDR into base address + mask
	ip, net, err := net.ParseCIDR(cidr)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Invalid CIDR %s: %s\n", cidr, err.Error())
		return
	}

	// Verify IPv4 for now
	ip4 := net.IP.To4()
	if ip4 == nil {
		fmt.Fprintf(os.Stderr, "Invalid IPv4 CIDR %s\n", cidr)
		return
	}

	net_base, err := utils.IPv4_to_UInt(net.IP.String())
	if err != nil {
		fmt.Fprintf(os.Stderr, "Invalid IPv4 Address %s: %s\n", ip.String(), err.Error())
		return
	}

	mask_ones, mask_total := net.Mask.Size()

	// Does not work for IPv6 due to cast to uint32
	net_size := uint32(math.Pow(2, float64(mask_total-mask_ones)))

	cur_base := net_base
	end_base := net_base + net_size

	var ndots uint32 = 3
	var block_size uint32 = 256

	// Handle massive network blocks
	if mask_ones <= 8 {
		ndots = 1
		block_size = 256 * 256 * 256
	} else if mask_ones <= 16 {
		ndots = 2
		block_size = 256 * 256
	}

	// Iterate by block size
	for ; end_base-cur_base >= block_size; cur_base += block_size {
		ip_prefix := strings.Join(strings.SplitN(utils.UInt_to_IPv4(cur_base), ".", 4)[0:ndots], ".") + "."
		searchPrefixIPv4(m, ip_prefix)
	}

	if end_base-cur_base == 0 {
		return
	}

	// Handle any leftovers by looking up a full /24 and ignoring stuff outside our range
	ip_prefix := strings.Join(strings.SplitN(utils.UInt_to_IPv4(cur_base), ".", 4)[0:3], ".") + "."

	it := mtbl.IterPrefix(m, []byte(ip_prefix))
	for {
		key_bytes, val_bytes, ok := it.Next()
		if !ok {
			break
		}

		// Only print results that are valid IPV4 addresses within our CIDR range
		cur_val, _ := utils.IPv4_to_UInt(string(key_bytes))
		if cur_val >= cur_base && cur_val <= end_base {
			if utils.Match_IPv4.Match(key_bytes) {
				writeOutput(key_bytes, val_bytes)
			}
		}
	}
}

func main() {

	runtime.GOMAXPROCS(runtime.NumCPU())
	os.Setenv("LC_ALL", "C")

	flag.Usage = func() { usage() }

	key_only = flag.Bool("k", false, "Display key names only")
	val_only = flag.Bool("v", false, "Display values only")
	prefix = flag.String("p", "", "Only return keys with this prefix")
	rev_prefix = flag.String("r", "", "Only return keys with this prefix in reverse form")
	rev_key = flag.Bool("R", false, "Display matches with the key in reverse form")
	no_quotes = flag.Bool("n", false, "Print raw values, not quoted values")
	as_json = flag.Bool("j", false, "Print each record as a single line of JSON")
	version = flag.Bool("version", false, "Show the version and build timestamp")
	domain = flag.String("domain", "", "Search for all matches for a specified domain")
	cidr = flag.String("cidr", "", "Search for all matches for the specified CIDR")

	flag.Parse()

	if *version {
		utils.PrintVersion()
		os.Exit(0)
	}

	if len(flag.Args()) == 0 {
		usage()
		os.Exit(1)
	}

	if *key_only && *val_only {
		fmt.Fprintf(os.Stderr, "Error: Only one of -k or -v can be specified\n")
		usage()
		os.Exit(1)
	}

	if len(*prefix) > 0 && len(*rev_prefix) > 0 {
		fmt.Fprintf(os.Stderr, "Error: Only one of -p or -r can be specified\n")
		usage()
		os.Exit(1)
	}

	if len(*domain) > 0 && (len(*prefix) > 0 || len(*rev_prefix) > 0 || len(*cidr) > 0) {
		fmt.Fprintf(os.Stderr, "Error: Only one of -p, -r, -domain, or -cidr can be specified\n")
		usage()
		os.Exit(1)
	}

	if len(*cidr) > 0 && (len(*prefix) > 0 || len(*rev_prefix) > 0 || len(*domain) > 0) {
		fmt.Fprintf(os.Stderr, "Error: Only one of -p, -r, -domain, or -cidr can be specified\n")
		usage()
		os.Exit(1)
	}

	paths := findPaths(flag.Args())

	m := mtbl.MergerInit(&mtbl.MergerOptions{Merge: mergeFunc})
	defer m.Destroy()

	for i := range paths {
		path := paths[i]

		r, e := mtbl.ReaderInit(path, &mtbl.ReaderOptions{VerifyChecksums: true})
		defer r.Destroy()
		if e != nil {
			fmt.Fprintf(os.Stderr, "Error reading %s: %s\n", path, e)
			os.Exit(1)
		}
		m.AddSource(r)
	}

	if len(*domain) > 0 {
		searchDomain(m, *domain)
		return
	}

	if len(*cidr) > 0 {
		searchCIDR(m, *cidr)
		return
	}

	if len(*prefix) > 0 {
		searchPrefix(m, *prefix)
		return
	}

	if len(*rev_prefix) > 0 {
		p := utils.ReverseKey(*rev_prefix)
		searchPrefix(m, p)
		return
	}

	searchAll(m)
}
<|endoftext|>


<commit_before>// Copyright © 2016 Nagy Károly Gábriel <karasz@jpi.io>
// This file, part of {{ .appName }}, is free and unencumbered
// software released into the public domain.
// For more information, please refer to <http://unlicense.org/>
//

package cmd

import (
	"fmt"

	"github.com/spf13/cobra"
)

var GVersion = "0.0.1"

// versionCmd represents the version command
var versionCmd = &cobra.Command{
	Use:   "version",
	Short: "Print the version number of genet",
	Long:  `All software has versions. This is genet's.`,
	Run: func(cmd *cobra.Command, args []string) {
		// TODO: Work your own magic here
		fmt.Printf("Genet's version is %s\n", GVersion)
	},
}

func init() {
	RootCmd.AddCommand(versionCmd)

}
<commit_msg>fixed version cmd to not have arguments<commit_after>// Copyright © 2016 Nagy Károly Gábriel <karasz@jpi.io>
// This file, part of {{ .appName }}, is free and unencumbered
// software released into the public domain.
// For more information, please refer to <http://unlicense.org/>
//

package cmd

import (
	"fmt"

	"github.com/spf13/cobra"
)

var GVersion = "0.0.1"

// versionCmd represents the version command
var versionCmd = &cobra.Command{
	Use:   "version",
	Short: "Print the version number of genet",
	Long:  `All software has versions. This is genet's.`,
	Run: func(cmd *cobra.Command, args []string) {
		fmt.Printf("Genet's version is %s\n", GVersion)
	},
}

func init() {
	versionCmd.SetUsageTemplate("Usage: \n\tgenet version\n\n")
	RootCmd.AddCommand(versionCmd)

}
<|endoftext|>


<commit_before>package cmd

import "github.com/spf13/cobra"

func init() {
	RootCmd.AddCommand(versionCmd)
}

var versionCmd = &cobra.Command{
	Use:   "version",
	Short: "Displays the version of mbt",
	Run: func(cmd *cobra.Command, args []string) {
		println("0.8.0")
	},
}
<commit_msg>Bump version<commit_after>package cmd

import "github.com/spf13/cobra"

func init() {
	RootCmd.AddCommand(versionCmd)
}

var versionCmd = &cobra.Command{
	Use:   "version",
	Short: "Displays the version of mbt",
	Run: func(cmd *cobra.Command, args []string) {
		println("0.8.1")
	},
}
<|endoftext|>


<commit_before>// package name: riffle
package main

import (
	"C"
	"unsafe"

	"github.com/exis-io/core"
	"github.com/exis-io/core/goRiffle"
)

// Required main method
func main() {}

// By default always connect to the production fabric at node.exis.io
var fabric string = core.FabricProduction

//export CBID
func CBID() uint {
	return core.NewID()
}

//export NewDomain
func NewDomain(name *C.char) unsafe.Pointer {
	d := core.NewDomain(C.GoString(name), nil)
	return unsafe.Pointer(&d)
}

//export Subdomain
func Subdomain(pdomain unsafe.Pointer, name *C.char) unsafe.Pointer {
	d := *(*core.Domain)(pdomain)
	n := d.Subdomain(C.GoString(name))
	return unsafe.Pointer(&n)
}

//export Receive
func Receive(pdomain unsafe.Pointer) []byte {
	d := *(*core.Domain)(pdomain)
	return []byte(core.MantleMarshall(d.GetApp().CallbackListen()))
}

//export Join
func Join(pdomain unsafe.Pointer, cb uint, eb uint) {
	d := *(*core.Domain)(pdomain)

	if c, err := goRiffle.Open(fabric); err != nil {
		d.GetApp().CallbackSend(eb, err.Error())
	} else {
		if err := d.Join(c); err != nil {
			d.GetApp().CallbackSend(eb, err.Error())
		} else {
			d.GetApp().CallbackSend(cb)
		}
	}
}

//export Subscribe
func Subscribe(pdomain unsafe.Pointer, cb uint, endpoint *C.char) {
	d := *(*core.Domain)(pdomain)
	go func() {
		d.Subscribe(C.GoString(endpoint), cb, make([]interface{}, 0))
	}()
}

//export Register
func Register(pdomain unsafe.Pointer, cb uint, endpoint *C.char) {
	d := *(*core.Domain)(pdomain)
	go func() {
		d.Register(C.GoString(endpoint), cb, make([]interface{}, 0))
	}()
}

//export Publish
func Publish(pdomain unsafe.Pointer, cb uint, endpoint *C.char, args *C.char) {
	d := *(*core.Domain)(pdomain)
    a := C.GoString(args)
    s := core.MantleUnmarshal(a)
    core.Debug("String: %s, Unmarshalled: %s", a, s)
	go func() {
		d.Publish(C.GoString(endpoint), cb, s)
	}()
}

//export Call
func Call(pdomain unsafe.Pointer, cb uint, endpoint *C.char, args *C.char) {
	d := *(*core.Domain)(pdomain)
	go func() {
		d.Call(C.GoString(endpoint), cb, core.MantleUnmarshal(C.GoString(args)))
	}()
}

//export Yield
func Yield(pdomain unsafe.Pointer, request uint, args *C.char) {
	d := *(*core.Domain)(pdomain)
	go func() {
		d.GetApp().Yield(request, core.MantleUnmarshal(C.GoString(args)))
	}()
}

//export Unsubscribe
func Unsubscribe(pdomain unsafe.Pointer, endpoint *C.char) {
	d := *(*core.Domain)(pdomain)
	go func() {
		d.Unsubscribe(C.GoString(endpoint))
	}()
}

//export Unregister
func Unregister(pdomain unsafe.Pointer, endpoint *C.char) {
	d := *(*core.Domain)(pdomain)
	go func() {
		d.Unregister(C.GoString(endpoint))
	}()
}

//export Leave
func Leave(pdomain unsafe.Pointer) {
	d := *(*core.Domain)(pdomain)
	go func() {
		d.Leave()
	}()
}

//export SetLogLevelOff
func SetLogLevelOff() { core.LogLevel = core.LogLevelOff }

//export SetLogLevelApp
func SetLogLevelApp() { core.LogLevel = core.LogLevelApp }

//export SetLogLevelErr
func SetLogLevelErr() { core.LogLevel = core.LogLevelErr }

//export SetLogLevelWarn
func SetLogLevelWarn() { core.LogLevel = core.LogLevelWarn }

//export SetLogLevelInfo
func SetLogLevelInfo() { core.LogLevel = core.LogLevelInfo }

//export SetLogLevelDebug
func SetLogLevelDebug() { core.LogLevel = core.LogLevelDebug }

//export SetFabricDev
func SetFabricDev() { fabric = core.FabricDev }

//export SetFabricSandbox
func SetFabricSandbox() { fabric = core.FabricSandbox }

//export SetFabricProduction
func SetFabricProduction() { fabric = core.FabricProduction }

//export SetFabricLocal
func SetFabricLocal() { fabric = core.FabricLocal }

//export MantleSetFabric
func MantleSetFabric(url *C.char) { fabric = C.GoString(url) }

//export Application
func Application(s string) { core.Application("%s", s) }

//export Debug
func Debug(s string) { core.Debug("%s", s) }

//export Info
func Info(s string) { core.Info("%s", s) }

//export Warn
func Warn(s string) { core.Warn("%s", s) }

//export Error
func Error(s string) { core.Error("%s", s) }
<commit_msg>logging<commit_after>// package name: riffle
package main

import (
	"C"
	"unsafe"

	"github.com/exis-io/core"
	"github.com/exis-io/core/goRiffle"
)

// Required main method
func main() {}

// By default always connect to the production fabric at node.exis.io
var fabric string = core.FabricProduction

//export CBID
func CBID() uint {
	return core.NewID()
}

//export NewDomain
func NewDomain(name *C.char) unsafe.Pointer {
	d := core.NewDomain(C.GoString(name), nil)
	return unsafe.Pointer(&d)
}

//export Subdomain
func Subdomain(pdomain unsafe.Pointer, name *C.char) unsafe.Pointer {
	d := *(*core.Domain)(pdomain)
	n := d.Subdomain(C.GoString(name))
	return unsafe.Pointer(&n)
}

//export Receive
func Receive(pdomain unsafe.Pointer) []byte {
	d := *(*core.Domain)(pdomain)
	return []byte(core.MantleMarshall(d.GetApp().CallbackListen()))
}

//export Join
func Join(pdomain unsafe.Pointer, cb uint, eb uint) {
	d := *(*core.Domain)(pdomain)

	if c, err := goRiffle.Open(fabric); err != nil {
		d.GetApp().CallbackSend(eb, err.Error())
	} else {
		if err := d.Join(c); err != nil {
			d.GetApp().CallbackSend(eb, err.Error())
		} else {
			d.GetApp().CallbackSend(cb)
		}
	}
}

//export Subscribe
func Subscribe(pdomain unsafe.Pointer, cb uint, endpoint *C.char) {
	d := *(*core.Domain)(pdomain)
	go func() {
		d.Subscribe(C.GoString(endpoint), cb, make([]interface{}, 0))
	}()
}

//export Register
func Register(pdomain unsafe.Pointer, cb uint, endpoint *C.char) {
	d := *(*core.Domain)(pdomain)
	go func() {
		d.Register(C.GoString(endpoint), cb, make([]interface{}, 0))
	}()
}

//export Publish
func Publish(pdomain unsafe.Pointer, cb uint, endpoint *C.char, args *C.char) {
	d := *(*core.Domain)(pdomain)
	a := C.GoString(args)
	s := core.MantleUnmarshal(a)
	core.Debug("String: %s, Unmarshalled: %s", a, s)
	go func() {
		d.Publish(C.GoString(endpoint), cb, s)
	}()
}

//export Call
func Call(pdomain unsafe.Pointer, cb uint, endpoint *C.char, args *C.char) {
	d := *(*core.Domain)(pdomain)
	go func() {
		d.Call(C.GoString(endpoint), cb, core.MantleUnmarshal(C.GoString(args)))
	}()
}

//export Yield
func Yield(pdomain unsafe.Pointer, request uint, args *C.char) {
	d := *(*core.Domain)(pdomain)
	go func() {
		d.GetApp().Yield(request, core.MantleUnmarshal(C.GoString(args)))
	}()
}

//export Unsubscribe
func Unsubscribe(pdomain unsafe.Pointer, endpoint *C.char) {
	d := *(*core.Domain)(pdomain)
	go func() {
		d.Unsubscribe(C.GoString(endpoint))
	}()
}

//export Unregister
func Unregister(pdomain unsafe.Pointer, endpoint *C.char) {
	d := *(*core.Domain)(pdomain)
	go func() {
		d.Unregister(C.GoString(endpoint))
	}()
}

//export Leave
func Leave(pdomain unsafe.Pointer) {
	d := *(*core.Domain)(pdomain)
	go func() {
		d.Leave()
	}()
}

//export SetLogLevelOff
func SetLogLevelOff() { core.LogLevel = core.LogLevelOff }

//export SetLogLevelApp
func SetLogLevelApp() { core.LogLevel = core.LogLevelApp }

//export SetLogLevelErr
func SetLogLevelErr() { core.LogLevel = core.LogLevelErr }

//export SetLogLevelWarn
func SetLogLevelWarn() { core.LogLevel = core.LogLevelWarn }

//export SetLogLevelInfo
func SetLogLevelInfo() { core.LogLevel = core.LogLevelInfo }

//export SetLogLevelDebug
func SetLogLevelDebug() { core.LogLevel = core.LogLevelDebug }

//export SetFabricDev
func SetFabricDev() { fabric = core.FabricDev }

//export SetFabricSandbox
func SetFabricSandbox() { fabric = core.FabricSandbox }

//export SetFabricProduction
func SetFabricProduction() { fabric = core.FabricProduction }

//export SetFabricLocal
func SetFabricLocal() { fabric = core.FabricLocal }

//export MantleSetFabric
func MantleSetFabric(url *C.char) { fabric = C.GoString(url) }

//export Application
func Application(s *C.char) { core.Application("%s", C.GoString(s)) }

//export Debug
func Debug(s *C.char) { core.Debug("%s", C.GoString(s)) }

//export Info
func Info(s *C.char) { core.Info("%s", C.GoString(s)) }

//export Warn
func Warn(s *C.char) { core.Warn("%s", C.GoString(s)) }

//export Error
func Error(s *C.char) { core.Error("%s", C.GoString(s)) }
<|endoftext|>