<commit_before>// Copyright 2014 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

/*
Git-codereview manages the code review process for Git changes using a Gerrit
server.

The git-codereview tool manages ``change branches'' in the local git repository.
Each such branch tracks a single commit, or ``pending change,''
that is reviewed using a Gerrit server; the Gerrit remote must be
named ``origin'' in the local git repo.

Modifications to the pending change are applied by amending the commit.
This process implements the ``single-commit feature branch'' model.
Creating multiple-commit feature branches, for example to break a large
change into a reviewable sequence, is also supported; see the discussion below.

Once installed as git-codereview, the tool's commands are available through git
either by running

	git codereview <command>

or, if aliases are installed, as

	git <command>

The review tool's command names do not conflict with any extant git commands.
This document uses the first form for clarity but most users install these
aliases in their .gitconfig file:

	[alias]
		change = codereview change
		gofmt = codereview gofmt
		mail = codereview mail
		pending = codereview pending
		rebase-work = codereview rebase-work
		submit = codereview submit
		sync = codereview sync

Single-Commit Work Branches

For simple, unrelated changes, the typical usage of the git-codereview tool
is to place each pending change in its own Git branch.
In this workflow, the work branch contains
either no pending change beyond origin/master (when there's no local work)
or exactly one pending change beyond origin/master (the change being developed).

When there is no pending change on the work branch,
``git codereview change'' creates one by running ``git commit.''
Otherwise, when there is already a pending change,
``git codereview change'' revises it by running ``git commit --amend.''

The ``git codereview mail'' and ``git codereview submit'' commands
implicitly operate on the lone pending change.

Multiple-Commit Work Branches

Of course, it is not always feasible to put each pending change in a separate branch.
A sequence of changes that build on one another is more easily
managed as multiple commits on a single branch, and the git-codereview tool
supports this workflow as well.
To add a new pending change, invoke ``git commit'' directly,
instead of ``git codereview change''.
The git-codereview tool adjusts its behavior when there are
multiple pending changes.

The ``git codereview change'' command amends the top commit in the stack (HEAD).
To amend a commit further down the stack, use Git's rebase support,
for example by using ``git commit --fixup'' followed by ``git codereview rebase-work.''

The ``git codereview mail'' command requires an explicit revision argument,
but note that since ``git codereview mail'' is implemented as a ``git push,''
any commits earlier in the stack are necessarily also mailed.

The ``git codereview submit'' command also requires an explicit revision argument,
and while earlier commits are necessarily still uploaded and mailed,
only the named revision or revisions are submitted (merged into origin/master).
In a single-commit work branch, a successful ``git codereview submit''
effectively runs ``git codereview sync'' automatically.
In a multiple-commit work branch, it does not, because
the implied ``git rebase'' may conflict with the remaining pending commits.
Instead it is necessary to run ``git codereview sync'' explicitly
(when ready) after ``git codereview submit.''

Reusing Work Branches

Although one common practice is to create a new branch for each pending change,
running ``git codereview submit'' (and possibly ``git codereview sync'')
leaves the current branch ready for reuse with a future change.
Some developers find it helpful to create a single work branch
(``git change work'') and then do all work in that branch,
possibly in the multiple-commit mode, never changing between branches.

Command Details

All commands accept these global flags:

The -v flag prints all commands that make changes.

The -n flag prints all commands that would be run, but does not run them.

Descriptions of each command follow.

Branchpoint

	git codereview branchpoint

The branchpoint command prints the commit hash of the most recent commit
on the current branch that is shared with the Gerrit server. This is the point
where local work branched from the published tree. The command is intended
mainly for use in scripts. For example, ``git diff $(git codereview branchpoint)''
or ``git log $(git codereview branchpoint)..HEAD''.

Change

The change command creates and moves between Git branches and maintains the
pending changes on work branches.

	git codereview change [-a] [-q] [branchname]

Given a branch name as an argument, the change command switches to the named
branch, creating it if necessary. If the branch is created and there are staged
changes, it will commit the changes to the branch, creating a new pending
change.

With no argument, the change command creates a new pending change from the
staged changes in the current branch or, if there is already a pending change,
amends that change.

The -q option skips the editing of an extant pending change's commit message.

The -a option automatically adds any unstaged edits in tracked files during
commit; it is equivalent to the 'git commit' -a option.

Gofmt

The gofmt command applies the gofmt program to all files modified in the
current work branch, both in the staging area (index) and the working tree
(local directory).

	git codereview gofmt [-l]

The -l option causes the command to list the files that need reformatting but
not reformat them. Otherwise, the gofmt command reformats modified files in
place. That is, files in the staging area are reformatted in the staging area,
and files in the working tree are reformatted in the working tree.

Help

The help command displays basic usage instructions.

	git codereview help

Hooks

The hooks command installs the Git hooks to enforce code review conventions.

	git codereview hooks

The pre-commit hook checks that all Go code is formatted with gofmt and that
the commit is not being made directly to the master branch.

The commit-msg hook adds the Gerrit ``Change-Id'' line to the commit message if
not present. It also checks that the message uses the convention established by
the Go project that the first line has the form, pkg/path: summary.

The hooks command will not overwrite an existing hook.
If it is not installing hooks, use ``git codereview hooks -v'' for details.
This hook installation is also done at startup by all other git codereview
commands, except ``git codereview help''.

Hook-Invoke

The hook-invoke command is an internal command that invokes the named Git hook.

	git codereview hook-invoke <hook> [args]

It is run by the shell scripts installed by the ``git codereview hooks'' command.

Mail

The mail command starts the code review process for the pending change.

	git codereview mail [-f] [-r email] [-cc email] [revision]

It pushes the pending change commit in the current branch to the Gerrit code
review server and prints the URL for the change on the server.
If the change already exists on the server, the mail command updates that
change with a new changeset.

The -r and -cc flags identify the email addresses of people to do the code
review and to be CC'ed about the code review.
Multiple addresses are given as a comma-separated list.

An email address passed to -r or -cc can be shortened from name@domain to name.
The mail command resolves such shortenings by reading the list of past reviewers
from the git repository log to find email addresses of the form name@somedomain
and then, in case of ambiguity, using the reviewer who appears most often.

The mail command fails if there are staged edits that are not committed.
The -f flag overrides this behavior.

The mail command updates the tag <branchname>.mailed to refer to the
commit that was most recently mailed, so running ``git diff <branchname>.mailed''
shows diffs between what is on the Gerrit server and the current directory.

If there are multiple pending commits, the revision argument is mandatory.
If no revision is specified, the mail command prints a short summary of
the pending commits for use in deciding which to mail.

Pending

The pending command prints to standard output the status of all pending changes
and staged, unstaged, and untracked files in the local repository.

	git codereview pending [-c] [-l] [-s]

The -c flag causes the command to show pending changes only on the current branch.

The -l flag causes the command to use only locally available information.
By default, it fetches recent commits and code review information from the
Gerrit server.

The -s flag causes the command to print abbreviated (short) output.

Common shorter aliases include ``git p'' for ``git pending''
and ``git pl'' for ``git pending -l'' (notably faster but without Gerrit information).

Rebase-work

The rebase-work command runs git rebase in interactive mode over pending changes.
It is shorthand for ``git rebase -i $(git codereview branchpoint)''.
It differs from plain ``git rebase -i'' in that the latter will try to incorporate
new commits from the origin branch during the rebase;
``git codereview rebase-work'' does not.

In multiple-commit workflows, rebase-work is used so often
that it can be helpful to alias it to ``git rw''.

Submit

The submit command pushes the pending change to the Gerrit server and tells
Gerrit to submit it to the master branch.

	git codereview submit [-i | revision...]

The command fails if there are modified files (staged or unstaged) that are not
part of the pending change.

The -i option causes the submit command to open a list of commits to submit
in the configured text editor, similar to ``git rebase -i''.

If multiple revisions are specified, the submit command submits each one in turn,
stopping at the first failure.

When run in a multiple-commit work branch,
either the -i option or the revision argument is mandatory.
If both are omitted, the submit command prints a short summary of
the pending commits for use in deciding which to submit.

After submitting the pending changes, the submit command tries to synchronize the
current branch to the submitted commit, if it can do so cleanly.
If not, it will prompt the user to run ``git codereview sync'' manually.

After a successful sync, the branch can be used to prepare a new change.

Sync

The sync command updates the local repository.

	git codereview sync

It fetches commits from the remote repository and merges them from the
upstream branch to the current branch, rebasing any pending changes.

Configuration

If a file named codereview.cfg is present in the repository root,
git-codereview will use it for configuration. It should contain lines
of this format:

	key: value

The ``gerrit'' key sets the Gerrit URL for this project. Git-codereview
automatically derives the Gerrit URL from repositories hosted in
*.googlesource.com. If not set or derived, the repository is assumed to
not have Gerrit, and certain features won't work.

The ``issuerepo'' key specifies the GitHub repository to use for issues, if
different from the source repository. If set to ``golang/go'', for example,
lines such as ``Fixes #123'' in a commit message will be rewritten to ``Fixes
golang/go#123''.

*/
package main
<commit_msg>git-codereview: document mail -trybot flag<commit_after>// Copyright 2014 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

/*
Git-codereview manages the code review process for Git changes using a Gerrit
server.

The git-codereview tool manages ``change branches'' in the local git repository.
Each such branch tracks a single commit, or ``pending change,''
that is reviewed using a Gerrit server; the Gerrit remote must be
named ``origin'' in the local git repo.

Modifications to the pending change are applied by amending the commit.
This process implements the ``single-commit feature branch'' model.
Creating multiple-commit feature branches, for example to break a large
change into a reviewable sequence, is also supported; see the discussion below.

Once installed as git-codereview, the tool's commands are available through git
either by running

	git codereview <command>

or, if aliases are installed, as

	git <command>

The review tool's command names do not conflict with any extant git commands.
This document uses the first form for clarity but most users install these
aliases in their .gitconfig file:

	[alias]
		change = codereview change
		gofmt = codereview gofmt
		mail = codereview mail
		pending = codereview pending
		rebase-work = codereview rebase-work
		submit = codereview submit
		sync = codereview sync

Single-Commit Work Branches

For simple, unrelated changes, the typical usage of the git-codereview tool
is to place each pending change in its own Git branch.
In this workflow, the work branch contains
either no pending change beyond origin/master (when there's no local work)
or exactly one pending change beyond origin/master (the change being developed).

When there is no pending change on the work branch,
``git codereview change'' creates one by running ``git commit.''
Otherwise, when there is already a pending change,
``git codereview change'' revises it by running ``git commit --amend.''

The ``git codereview mail'' and ``git codereview submit'' commands
implicitly operate on the lone pending change.

Multiple-Commit Work Branches

Of course, it is not always feasible to put each pending change in a separate branch.
A sequence of changes that build on one another is more easily
managed as multiple commits on a single branch, and the git-codereview tool
supports this workflow as well.
To add a new pending change, invoke ``git commit'' directly,
instead of ``git codereview change''.
The git-codereview tool adjusts its behavior when there are
multiple pending changes.

The ``git codereview change'' command amends the top commit in the stack (HEAD).
To amend a commit further down the stack, use Git's rebase support,
for example by using ``git commit --fixup'' followed by ``git codereview rebase-work.''

The ``git codereview mail'' command requires an explicit revision argument,
but note that since ``git codereview mail'' is implemented as a ``git push,''
any commits earlier in the stack are necessarily also mailed.

The ``git codereview submit'' command also requires an explicit revision argument,
and while earlier commits are necessarily still uploaded and mailed,
only the named revision or revisions are submitted (merged into origin/master).
In a single-commit work branch, a successful ``git codereview submit''
effectively runs ``git codereview sync'' automatically.
In a multiple-commit work branch, it does not, because
the implied ``git rebase'' may conflict with the remaining pending commits.
Instead it is necessary to run ``git codereview sync'' explicitly
(when ready) after ``git codereview submit.''

Reusing Work Branches

Although one common practice is to create a new branch for each pending change,
running ``git codereview submit'' (and possibly ``git codereview sync'')
leaves the current branch ready for reuse with a future change.
Some developers find it helpful to create a single work branch
(``git change work'') and then do all work in that branch,
possibly in the multiple-commit mode, never changing between branches.

Command Details

All commands accept these global flags:

The -v flag prints all commands that make changes.

The -n flag prints all commands that would be run, but does not run them.

Descriptions of each command follow.

Branchpoint

	git codereview branchpoint

The branchpoint command prints the commit hash of the most recent commit
on the current branch that is shared with the Gerrit server. This is the point
where local work branched from the published tree. The command is intended
mainly for use in scripts. For example, ``git diff $(git codereview branchpoint)''
or ``git log $(git codereview branchpoint)..HEAD''.

Change

The change command creates and moves between Git branches and maintains the
pending changes on work branches.

	git codereview change [-a] [-q] [branchname]

Given a branch name as an argument, the change command switches to the named
branch, creating it if necessary. If the branch is created and there are staged
changes, it will commit the changes to the branch, creating a new pending
change.

With no argument, the change command creates a new pending change from the
staged changes in the current branch or, if there is already a pending change,
amends that change.

The -q option skips the editing of an extant pending change's commit message.

The -a option automatically adds any unstaged edits in tracked files during
commit; it is equivalent to the 'git commit' -a option.

Gofmt

The gofmt command applies the gofmt program to all files modified in the
current work branch, both in the staging area (index) and the working tree
(local directory).

	git codereview gofmt [-l]

The -l option causes the command to list the files that need reformatting but
not reformat them. Otherwise, the gofmt command reformats modified files in
place. That is, files in the staging area are reformatted in the staging area,
and files in the working tree are reformatted in the working tree.

Help

The help command displays basic usage instructions.

	git codereview help

Hooks

The hooks command installs the Git hooks to enforce code review conventions.

	git codereview hooks

The pre-commit hook checks that all Go code is formatted with gofmt and that
the commit is not being made directly to the master branch.

The commit-msg hook adds the Gerrit ``Change-Id'' line to the commit message if
not present. It also checks that the message uses the convention established by
the Go project that the first line has the form, pkg/path: summary.

The hooks command will not overwrite an existing hook.
If it is not installing hooks, use ``git codereview hooks -v'' for details.
This hook installation is also done at startup by all other git codereview
commands, except ``git codereview help''.

Hook-Invoke

The hook-invoke command is an internal command that invokes the named Git hook.

	git codereview hook-invoke <hook> [args]

It is run by the shell scripts installed by the ``git codereview hooks'' command.

Mail

The mail command starts the code review process for the pending change.

	git codereview mail [-f] [-r email] [-cc email] [-trybot] [revision]

It pushes the pending change commit in the current branch to the Gerrit code
review server and prints the URL for the change on the server.
If the change already exists on the server, the mail command updates that
change with a new changeset.

The -r and -cc flags identify the email addresses of people to do the code
review and to be CC'ed about the code review.
Multiple addresses are given as a comma-separated list.

An email address passed to -r or -cc can be shortened from name@domain to name.
The mail command resolves such shortenings by reading the list of past reviewers
from the git repository log to find email addresses of the form name@somedomain
and then, in case of ambiguity, using the reviewer who appears most often.

The -trybot flag runs the trybots on all new or updated changes. It is
equivalent to setting the Run-Trybot+1 label from Gerrit.

The mail command fails if there are staged edits that are not committed.
The -f flag overrides this behavior.

The mail command updates the tag <branchname>.mailed to refer to the
commit that was most recently mailed, so running ``git diff <branchname>.mailed''
shows diffs between what is on the Gerrit server and the current directory.

If there are multiple pending commits, the revision argument is mandatory.
If no revision is specified, the mail command prints a short summary of
the pending commits for use in deciding which to mail.

Pending

The pending command prints to standard output the status of all pending changes
and staged, unstaged, and untracked files in the local repository.

	git codereview pending [-c] [-l] [-s]

The -c flag causes the command to show pending changes only on the current branch.

The -l flag causes the command to use only locally available information.
By default, it fetches recent commits and code review information from the
Gerrit server.

The -s flag causes the command to print abbreviated (short) output.

Common shorter aliases include ``git p'' for ``git pending''
and ``git pl'' for ``git pending -l'' (notably faster but without Gerrit information).

Rebase-work

The rebase-work command runs git rebase in interactive mode over pending changes.
It is shorthand for ``git rebase -i $(git codereview branchpoint)''.
It differs from plain ``git rebase -i'' in that the latter will try to incorporate
new commits from the origin branch during the rebase;
``git codereview rebase-work'' does not.

In multiple-commit workflows, rebase-work is used so often
that it can be helpful to alias it to ``git rw''.

Submit

The submit command pushes the pending change to the Gerrit server and tells
Gerrit to submit it to the master branch.

	git codereview submit [-i | revision...]

The command fails if there are modified files (staged or unstaged) that are not
part of the pending change.

The -i option causes the submit command to open a list of commits to submit
in the configured text editor, similar to ``git rebase -i''.

If multiple revisions are specified, the submit command submits each one in turn,
stopping at the first failure.

When run in a multiple-commit work branch,
either the -i option or the revision argument is mandatory.
If both are omitted, the submit command prints a short summary of
the pending commits for use in deciding which to submit.

After submitting the pending changes, the submit command tries to synchronize the
current branch to the submitted commit, if it can do so cleanly.
If not, it will prompt the user to run ``git codereview sync'' manually.

After a successful sync, the branch can be used to prepare a new change.

Sync

The sync command updates the local repository.

	git codereview sync

It fetches commits from the remote repository and merges them from the
upstream branch to the current branch, rebasing any pending changes.

Configuration

If a file named codereview.cfg is present in the repository root,
git-codereview will use it for configuration. It should contain lines
of this format:

	key: value

The ``gerrit'' key sets the Gerrit URL for this project. Git-codereview
automatically derives the Gerrit URL from repositories hosted in
*.googlesource.com. If not set or derived, the repository is assumed to
not have Gerrit, and certain features won't work.

The ``issuerepo'' key specifies the GitHub repository to use for issues, if
different from the source repository. If set to ``golang/go'', for example,
lines such as ``Fixes #123'' in a commit message will be rewritten to ``Fixes
golang/go#123''.

*/
package main
<|endoftext|>


<commit_before>package main

import (
	"fmt"
	"io"
	"os"

	"github.com/go-hep/hepmc"
)

func main() {
	var err error
	var r io.Reader
	var w io.Writer = os.Stdout

	switch len(os.Args) {
	case 1:
		r = os.Stdin
	case 2:
		r, err = os.Open(os.Args[1])
		if err != nil {
			fmt.Printf("**error: %v\n", err)
			os.Exit(1)
		}
	default:
	}

	dec := hepmc.NewDecoder(r)
	if dec == nil {
		fmt.Printf("**error: nil decoder\n")
		os.Exit(1)
	}

	for {
		var evt hepmc.Event
		err = dec.Decode(&evt)
		if err == io.EOF {
			break
		}
		if err != nil {
			fmt.Printf("**error: %v\n", err)
			os.Exit(1)
		}
		err = evt.Print(w)
		if err != nil {
			fmt.Printf("**error: %v\n", err)
			os.Exit(1)
		}
	}
}

// EOF
<commit_msg>go-hepmc-dump: add help<commit_after>// go-hepmc-dump is a simple command to dump in an almost human-friendly format
// the content of a hepmc file.
// ex:
//  $ go-hepmc-dump foo.hepmc | head -n20
// ________________________________________________________________________________
// GenEvent: #0000 ID=  111 SignalProcessGenVertex Barcode: 0
//  Momentum units:      GEV     Position units:       MM
//  Cross Section: 2.666668e+12 +/- 2.666668e+12
//  Entries this event: 129 vertices, 241 particles.
//  Beam Particle barcodes: 1 2
//  RndmState(0)=
//  Wgts(1)=(0,1.000000)
//  EventScale 1.438780e+00 [energy] 	 alphaQCD=4.519907e-01	 alphaQED=7.472465e-03
//                                     GenParticle Legend
//         Barcode   PDG ID      ( Px,       Py,       Pz,     E ) Stat  DecayVtx
// ________________________________________________________________________________
// GenVertex:       -1 ID:    0 (X,cT):0
//  I: 1         7        21 -5.87e-14, 9.60e-15, 3.41e+01, 3.41e+01 42        -1
//  O: 1         3        21  0.00e+00, 0.00e+00, 3.41e+01, 3.41e+01 21        -3
// GenVertex:       -2 ID:    0 (X,cT):0
//  I: 1         8        21  3.20e-14, 0.00e+00,-7.23e+01, 7.23e+01 41        -2
//  O: 2         4        21  0.00e+00, 0.00e+00,-1.19e+00, 1.19e+00 21        -3
//              11        21  9.42e-01,-1.56e-01,-7.11e+01, 7.11e+01 43       -12
//
package main

import (
	"fmt"
	"io"
	"os"

	"github.com/go-hep/hepmc"
)

func main() {
	var err error
	var r io.Reader
	var w io.Writer = os.Stdout

	switch len(os.Args) {
	case 1:
		r = os.Stdin
	case 2:
		r, err = os.Open(os.Args[1])
		if err != nil {
			fmt.Printf("**error: %v\n", err)
			os.Exit(1)
		}
	default:
	}

	dec := hepmc.NewDecoder(r)
	if dec == nil {
		fmt.Printf("**error: nil decoder\n")
		os.Exit(1)
	}

	for {
		var evt hepmc.Event
		err = dec.Decode(&evt)
		if err == io.EOF {
			break
		}
		if err != nil {
			fmt.Printf("**error: %v\n", err)
			os.Exit(1)
		}
		err = evt.Print(w)
		if err != nil {
			fmt.Printf("**error: %v\n", err)
			os.Exit(1)
		}
	}
}

// EOF
<|endoftext|>


<commit_before>package comment

/*
	Common implementation of a comment.
*/

import (
	"bytes"
	"encoding/gob"
	"encoding/json"
	"time"
)

// Comment is a struct containing a comment by a user on a given entity.
type Comment struct {
	Id        string
	Message   string
	Timestamp time.Time
	User      string
}

// Return a Comment instance.
func New(id, msg, user string) *Comment {
	return &Comment{
		Id:        id,
		Message:   msg,
		Timestamp: time.Now(),
		User:      user,
	}
}

// Return a copy of the Comment.
func (c *Comment) Copy() *Comment {
	return &Comment{
		Id:        c.Id,
		Message:   c.Message,
		Timestamp: c.Timestamp,
		User:      c.User,
	}
}

// comment makes up the raw contents of a Comment.
type comment struct {
	Id        string    `json:"id"`
	Message   string    `json:"message"`
	Timestamp time.Time `json:"timestamp"`
	User      string    `json:"user"`
}

// See docs for json.Marshaler.
func (c *Comment) MarshalJSON() ([]byte, error) {
	c.Timestamp = c.Timestamp.UTC()
	return json.Marshal(comment{
		Id:        c.Id,
		Message:   c.Message,
		Timestamp: c.Timestamp,
		User:      c.User,
	})
}

// See docs for json.Unmarshaler.
func (c *Comment) UnmarshalJSON(b []byte) error {
	var body comment
	if err := json.Unmarshal(b, &body); err != nil {
		return err
	}
	c.Id = body.Id
	c.Message = body.Message
	c.Timestamp = body.Timestamp.UTC()
	c.User = body.User
	return nil
}

// See docs for gob.GobEncoder.
func (c *Comment) GobEncode() ([]byte, error) {
	c.Timestamp = c.Timestamp.UTC()
	buf := bytes.NewBuffer([]byte{})
	if err := gob.NewEncoder(buf).Encode(comment{
		Id:        c.Id,
		Message:   c.Message,
		Timestamp: c.Timestamp,
		User:      c.User,
	}); err != nil {
		return nil, err
	}
	return buf.Bytes(), nil
}

// See docs for gob.GobDecoder.
func (c *Comment) GobDecode(b []byte) error {
	var body comment
	if err := gob.NewDecoder(bytes.NewReader(b)).Decode(&body); err != nil {
		return err
	}
	c.Id = body.Id
	c.Message = body.Message
	c.Timestamp = body.Timestamp.UTC()
	c.User = body.User
	return nil
}
<commit_msg>Address comment from https://skia-review.googlesource.com/c/28982<commit_after>package comment

/*
	Common implementation of a comment.
*/

import (
	"bytes"
	"encoding/gob"
	"encoding/json"
	"time"
)

// Comment is a struct containing a comment by a user on a given entity.
type Comment struct {
	Id        string
	Message   string
	Timestamp time.Time
	User      string
}

// Return a Comment instance.
func New(id, msg, user string) *Comment {
	return &Comment{
		Id:        id,
		Message:   msg,
		Timestamp: time.Now().UTC(),
		User:      user,
	}
}

// Return a copy of the Comment.
func (c *Comment) Copy() *Comment {
	return &Comment{
		Id:        c.Id,
		Message:   c.Message,
		Timestamp: c.Timestamp,
		User:      c.User,
	}
}

// comment makes up the raw contents of a Comment.
type comment struct {
	Id        string    `json:"id"`
	Message   string    `json:"message"`
	Timestamp time.Time `json:"timestamp"`
	User      string    `json:"user"`
}

// See docs for json.Marshaler.
func (c *Comment) MarshalJSON() ([]byte, error) {
	c.Timestamp = c.Timestamp.UTC()
	return json.Marshal(comment{
		Id:        c.Id,
		Message:   c.Message,
		Timestamp: c.Timestamp,
		User:      c.User,
	})
}

// See docs for json.Unmarshaler.
func (c *Comment) UnmarshalJSON(b []byte) error {
	var body comment
	if err := json.Unmarshal(b, &body); err != nil {
		return err
	}
	c.Id = body.Id
	c.Message = body.Message
	c.Timestamp = body.Timestamp.UTC()
	c.User = body.User
	return nil
}

// See docs for gob.GobEncoder.
func (c *Comment) GobEncode() ([]byte, error) {
	c.Timestamp = c.Timestamp.UTC()
	buf := bytes.NewBuffer([]byte{})
	if err := gob.NewEncoder(buf).Encode(comment{
		Id:        c.Id,
		Message:   c.Message,
		Timestamp: c.Timestamp,
		User:      c.User,
	}); err != nil {
		return nil, err
	}
	return buf.Bytes(), nil
}

// See docs for gob.GobDecoder.
func (c *Comment) GobDecode(b []byte) error {
	var body comment
	if err := gob.NewDecoder(bytes.NewReader(b)).Decode(&body); err != nil {
		return err
	}
	c.Id = body.Id
	c.Message = body.Message
	c.Timestamp = body.Timestamp.UTC()
	c.User = body.User
	return nil
}
<|endoftext|>


<commit_before>package launchd

import (
	"bufio"
	"bytes"
	"fmt"
	"io/ioutil"
	"os"
	"os/exec"
	"os/user"
	"path/filepath"
	"strconv"
	"strings"

	"github.com/keybase/client/go/libkb"
)

var log = libkb.G.Log

// Service defines a service
type Service struct {
	label string
}

// NewService constructs a launchd service.
func NewService(label string) Service {
	return Service{
		label: label,
	}
}

// Label for service
func (s Service) Label() string { return s.label }

// Plist defines a launchd plist
type Plist struct {
	label      string
	binPath    string
	args       []string
	envVars    map[string]string
	workingDir string
}

// NewPlist constructs a launchd service.
func NewPlist(label string, binPath string, args []string, envVars map[string]string, workingDir string) Plist {
	return Plist{
		label:      label,
		binPath:    binPath,
		args:       args,
		envVars:    envVars,
		workingDir: workingDir,
	}
}

// Load will load the service.
// If restart=true, then we'll unload it first.
func (s Service) Load(restart bool) error {
	// Unload first if we're forcing
	plistDest := s.plistDestination()
	if restart {
		exec.Command("/bin/launchctl", "unload", plistDest).Output()
	}
	log.Info("Loading %s", s.label)
	_, err := exec.Command("/bin/launchctl", "load", "-w", plistDest).Output()
	return err
}

// Unload will unload the service
func (s Service) Unload() error {
	plistDest := s.plistDestination()
	log.Info("Unloading %s", s.label)
	_, err := exec.Command("/bin/launchctl", "unload", plistDest).Output()
	return err
}

// Install will install the launchd service
func (s Service) Install(p Plist) (err error) {
	if _, err := os.Stat(p.binPath); os.IsNotExist(err) {
		return err
	}
	plist := p.plist()
	plistDest := s.plistDestination()

	log.Info("Saving %s", plistDest)
	file := libkb.NewFile(plistDest, []byte(plist), 0644)
	err = file.Save()
	if err != nil {
		return
	}

	err = s.Load(true)
	return
}

// Uninstall will uninstall the launchd service
func (s Service) Uninstall() (err error) {
	err = s.Unload()
	if err != nil {
		return
	}

	plistDest := s.plistDestination()
	if _, err := os.Stat(plistDest); err == nil {
		log.Info("Removing %s", plistDest)
		err = os.Remove(plistDest)
	}
	return
}

// ListServices will return service with label containing the filter string.
func ListServices(filter string) ([]Service, error) {
	files, err := ioutil.ReadDir(launchAgentDir())
	if err != nil {
		return nil, err
	}
	var services []Service
	for _, f := range files {
		name := f.Name()
		suffix := ".plist"
		// We care about services that contain the filter word and end in .plist
		if strings.Contains(name, filter) && strings.HasSuffix(name, suffix) {
			label := name[0 : len(name)-len(suffix)]
			service := NewService(label)
			services = append(services, service)
		}
	}
	return services, nil
}

// ServiceStatus defines status for a service
type ServiceStatus struct {
	label          string
	pid            string // May be blank if not set, or a number "123"
	lastExitStatus string // Will be blank if pid > 0, or a number "123"
}

// Label for status
func (s ServiceStatus) Label() string { return s.label }

// Pid for status (empty string if not running)
func (s ServiceStatus) Pid() string { return s.pid }

// Description returns service status info
func (s ServiceStatus) Description() string {
	var status string
	infos := []string{}
	if s.IsRunning() {
		status = "Running"
		infos = append(infos, fmt.Sprintf("(pid=%s)", s.pid))
	} else {
		status = "Not Running"
	}
	if s.lastExitStatus != "" {
		infos = append(infos, fmt.Sprintf("exit=%s", s.lastExitStatus))
	}
	return status + " " + strings.Join(infos, ", ")
}

// IsRunning is true if the service is running (with a pid)
func (s ServiceStatus) IsRunning() bool {
	return s.pid != ""
}

// StatusDescription returns the service status description
func (s Service) StatusDescription() string {
	status, err := s.Status()
	if err != nil {
		return fmt.Sprintf("%s: %v", s.label, err)
	}
	return fmt.Sprintf("%s: %s", s.label, status.Description())
}

// Status returns service status
func (s Service) Status() (*ServiceStatus, error) {
	out, err := exec.Command("/bin/launchctl", "list").Output()
	if err != nil {
		return nil, err
	}

	var pid, lastExitStatus string
	var found bool
	scanner := bufio.NewScanner(bytes.NewBuffer(out))
	for scanner.Scan() {
		line := scanner.Text()
		fields := strings.Fields(line)
		if len(fields) == 3 && fields[2] == s.label {
			found = true
			if fields[0] != "-" {
				pid = fields[0]
			}
			if fields[1] != "-" {
				lastExitStatus = fields[1]
			}
		}
	}

	if found {
		// If pid is set and > 0, then clear lastExitStatus which is the
		// exit status of the previous run and doesn't mean anything for
		// the current state. Clearing it to avoid confusion.
		pidInt, _ := strconv.ParseInt(pid, 0, 64)
		if pid != "" && pidInt > 0 {
			lastExitStatus = ""
		}
		return &ServiceStatus{label: s.label, pid: pid, lastExitStatus: lastExitStatus}, nil
	}

	return nil, nil
}

// ShowServices ouputs keybase service info
func ShowServices(filter string) (err error) {
	services, err := ListServices(filter)
	if err != nil {
		return
	}
	if len(services) > 0 {
		log.Info("Found %s:", libkb.Pluralize(len(services), "service", "services"))
		for _, service := range services {
			log.Info(service.StatusDescription())
		}
	} else {
		log.Info("No services")
	}
	return
}

// Install will install a service
func Install(plist Plist) (err error) {
	service := NewService(plist.label)
	return service.Install(plist)
}

// Uninstall will uninstall a keybase service
func Uninstall(label string) error {
	service := NewService(label)
	return service.Uninstall()
}

// Start will start a keybase service
func Start(label string) error {
	service := NewService(label)
	return service.Load(false)
}

// Stop will stop a keybase service
func Stop(label string) error {
	service := NewService(label)
	return service.Unload()
}

// ShowStatus shows status info for a service
func ShowStatus(label string) error {
	service := NewService(label)
	status, err := service.Status()
	if err != nil {
		return err
	}
	log.Info(status.Description())
	return nil
}

// Restart restarts a service
func Restart(label string) error {
	service := NewService(label)
	return service.Load(true)
}

func launchAgentDir() string {
	return filepath.Join(launchdHomeDir(), "Library", "LaunchAgents")
}

func (s Service) plistDestination() string {
	return filepath.Join(launchAgentDir(), s.label+".plist")
}

func launchdHomeDir() string {
	currentUser, err := user.Current()
	if err != nil {
		panic(err)
	}
	return currentUser.HomeDir
}

func launchdLogDir() string {
	return filepath.Join(launchdHomeDir(), "Library", "Logs")
}

// TODO Use go-plist library
func (p Plist) plist() string {
	logFile := filepath.Join(launchdLogDir(), p.label+".log")

	encodeTag := func(name, val string) string {
		return fmt.Sprintf("<%s>%s</%s>", name, val, name)
	}

	pargs := []string{}
	// First arg is the keybase executable
	pargs = append(pargs, encodeTag("string", p.binPath))
	for _, arg := range p.args {
		pargs = append(pargs, encodeTag("string", arg))
	}

	envVars := []string{}
	for key, value := range p.envVars {
		envVars = append(envVars, encodeTag("key", key))
		envVars = append(envVars, encodeTag("string", value))
	}

	return `<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>` + p.label + `</string>
  <key>EnvironmentVariables</key>
  <dict>` + "\n    " + strings.Join(envVars, "\n    ") + `
  </dict>
  <key>ProgramArguments</key>
  <array>` + "\n    " + strings.Join(pargs, "\n    ") + `
  </array>
  <key>KeepAlive</key>
  <true/>
  <key>RunAtLoad</key>
  <true/>
  <key>WorkingDirectory</key>
  <string>` + p.workingDir + `</string>
  <key>StandardErrorPath</key>
  <string>` + logFile + `</string>
  <key>StandardOutPath</key>
  <string>` + logFile + `</string>
</dict>
</plist>`
}
<commit_msg>Launchd install ensure cwd exists<commit_after>package launchd

import (
	"bufio"
	"bytes"
	"fmt"
	"io/ioutil"
	"os"
	"os/exec"
	"os/user"
	"path/filepath"
	"strconv"
	"strings"

	"github.com/keybase/client/go/libkb"
)

var log = libkb.G.Log

// Service defines a service
type Service struct {
	label string
}

// NewService constructs a launchd service.
func NewService(label string) Service {
	return Service{
		label: label,
	}
}

// Label for service
func (s Service) Label() string { return s.label }

// Plist defines a launchd plist
type Plist struct {
	label      string
	binPath    string
	args       []string
	envVars    map[string]string
	workingDir string
}

// NewPlist constructs a launchd service.
func NewPlist(label string, binPath string, args []string, envVars map[string]string, workingDir string) Plist {
	return Plist{
		label:      label,
		binPath:    binPath,
		args:       args,
		envVars:    envVars,
		workingDir: workingDir,
	}
}

// Load will load the service.
// If restart=true, then we'll unload it first.
func (s Service) Load(restart bool) error {
	// Unload first if we're forcing
	plistDest := s.plistDestination()
	if restart {
		exec.Command("/bin/launchctl", "unload", plistDest).Output()
	}
	log.Info("Loading %s", s.label)
	_, err := exec.Command("/bin/launchctl", "load", "-w", plistDest).Output()
	return err
}

// Unload will unload the service
func (s Service) Unload() error {
	plistDest := s.plistDestination()
	log.Info("Unloading %s", s.label)
	_, err := exec.Command("/bin/launchctl", "unload", plistDest).Output()
	return err
}

// Install will install the launchd service
func (s Service) Install(p Plist) (err error) {
	if _, err := os.Stat(p.binPath); os.IsNotExist(err) {
		return err
	}
	plist := p.plist()
	plistDest := s.plistDestination()

	log.Info("Saving %s", plistDest)
	file := libkb.NewFile(plistDest, []byte(plist), 0644)
	err = file.Save()
	if err != nil {
		return
	}

	err = s.Load(true)
	return
}

// Uninstall will uninstall the launchd service
func (s Service) Uninstall() (err error) {
	err = s.Unload()
	if err != nil {
		return
	}

	plistDest := s.plistDestination()
	if _, err := os.Stat(plistDest); err == nil {
		log.Info("Removing %s", plistDest)
		err = os.Remove(plistDest)
	}
	return
}

// ListServices will return service with label containing the filter string.
func ListServices(filter string) ([]Service, error) {
	files, err := ioutil.ReadDir(launchAgentDir())
	if err != nil {
		return nil, err
	}
	var services []Service
	for _, f := range files {
		name := f.Name()
		suffix := ".plist"
		// We care about services that contain the filter word and end in .plist
		if strings.Contains(name, filter) && strings.HasSuffix(name, suffix) {
			label := name[0 : len(name)-len(suffix)]
			service := NewService(label)
			services = append(services, service)
		}
	}
	return services, nil
}

// ServiceStatus defines status for a service
type ServiceStatus struct {
	label          string
	pid            string // May be blank if not set, or a number "123"
	lastExitStatus string // Will be blank if pid > 0, or a number "123"
}

// Label for status
func (s ServiceStatus) Label() string { return s.label }

// Pid for status (empty string if not running)
func (s ServiceStatus) Pid() string { return s.pid }

// Description returns service status info
func (s ServiceStatus) Description() string {
	var status string
	infos := []string{}
	if s.IsRunning() {
		status = "Running"
		infos = append(infos, fmt.Sprintf("(pid=%s)", s.pid))
	} else {
		status = "Not Running"
	}
	if s.lastExitStatus != "" {
		infos = append(infos, fmt.Sprintf("exit=%s", s.lastExitStatus))
	}
	return status + " " + strings.Join(infos, ", ")
}

// IsRunning is true if the service is running (with a pid)
func (s ServiceStatus) IsRunning() bool {
	return s.pid != ""
}

// StatusDescription returns the service status description
func (s Service) StatusDescription() string {
	status, err := s.Status()
	if err != nil {
		return fmt.Sprintf("%s: %v", s.label, err)
	}
	return fmt.Sprintf("%s: %s", s.label, status.Description())
}

// Status returns service status
func (s Service) Status() (*ServiceStatus, error) {
	out, err := exec.Command("/bin/launchctl", "list").Output()
	if err != nil {
		return nil, err
	}

	var pid, lastExitStatus string
	var found bool
	scanner := bufio.NewScanner(bytes.NewBuffer(out))
	for scanner.Scan() {
		line := scanner.Text()
		fields := strings.Fields(line)
		if len(fields) == 3 && fields[2] == s.label {
			found = true
			if fields[0] != "-" {
				pid = fields[0]
			}
			if fields[1] != "-" {
				lastExitStatus = fields[1]
			}
		}
	}

	if found {
		// If pid is set and > 0, then clear lastExitStatus which is the
		// exit status of the previous run and doesn't mean anything for
		// the current state. Clearing it to avoid confusion.
		pidInt, _ := strconv.ParseInt(pid, 0, 64)
		if pid != "" && pidInt > 0 {
			lastExitStatus = ""
		}
		return &ServiceStatus{label: s.label, pid: pid, lastExitStatus: lastExitStatus}, nil
	}

	return nil, nil
}

// ShowServices ouputs keybase service info
func ShowServices(filter string) (err error) {
	services, err := ListServices(filter)
	if err != nil {
		return
	}
	if len(services) > 0 {
		log.Info("Found %s:", libkb.Pluralize(len(services), "service", "services"))
		for _, service := range services {
			log.Info(service.StatusDescription())
		}
	} else {
		log.Info("No services")
	}
	return
}

// Install will install a service
func Install(plist Plist) (err error) {
	service := NewService(plist.label)

	if plist.workingDir != "" {
		err := ensureDirectoryExists(plist.workingDir)
		if err != nil {
			return err
		}
	}

	return service.Install(plist)
}

// Uninstall will uninstall a keybase service
func Uninstall(label string) error {
	service := NewService(label)
	return service.Uninstall()
}

// Start will start a keybase service
func Start(label string) error {
	service := NewService(label)
	return service.Load(false)
}

// Stop will stop a keybase service
func Stop(label string) error {
	service := NewService(label)
	return service.Unload()
}

// ShowStatus shows status info for a service
func ShowStatus(label string) error {
	service := NewService(label)
	status, err := service.Status()
	if err != nil {
		return err
	}
	log.Info(status.Description())
	return nil
}

// Restart restarts a service
func Restart(label string) error {
	service := NewService(label)
	return service.Load(true)
}

func launchAgentDir() string {
	return filepath.Join(launchdHomeDir(), "Library", "LaunchAgents")
}

func (s Service) plistDestination() string {
	return filepath.Join(launchAgentDir(), s.label+".plist")
}

func launchdHomeDir() string {
	currentUser, err := user.Current()
	if err != nil {
		panic(err)
	}
	return currentUser.HomeDir
}

func launchdLogDir() string {
	return filepath.Join(launchdHomeDir(), "Library", "Logs")
}

func ensureDirectoryExists(dir string) error {
	if _, err := os.Stat(dir); os.IsNotExist(err) {
		err = os.MkdirAll(dir, 0700)
		return err
	}
	return nil
}

// TODO Use go-plist library
func (p Plist) plist() string {
	logFile := filepath.Join(launchdLogDir(), p.label+".log")

	encodeTag := func(name, val string) string {
		return fmt.Sprintf("<%s>%s</%s>", name, val, name)
	}

	pargs := []string{}
	// First arg is the keybase executable
	pargs = append(pargs, encodeTag("string", p.binPath))
	for _, arg := range p.args {
		pargs = append(pargs, encodeTag("string", arg))
	}

	envVars := []string{}
	for key, value := range p.envVars {
		envVars = append(envVars, encodeTag("key", key))
		envVars = append(envVars, encodeTag("string", value))
	}

	return `<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>` + p.label + `</string>
  <key>EnvironmentVariables</key>
  <dict>` + "\n    " + strings.Join(envVars, "\n    ") + `
  </dict>
  <key>ProgramArguments</key>
  <array>` + "\n    " + strings.Join(pargs, "\n    ") + `
  </array>
  <key>KeepAlive</key>
  <true/>
  <key>RunAtLoad</key>
  <true/>
  <key>WorkingDirectory</key>
  <string>` + p.workingDir + `</string>
  <key>StandardErrorPath</key>
  <string>` + logFile + `</string>
  <key>StandardOutPath</key>
  <string>` + logFile + `</string>
</dict>
</plist>`
}
<|endoftext|>


<commit_before>package pserver

import (
	"context"
	"errors"
	"fmt"
	"strconv"
	"strings"
	"sync"
	"time"

	"github.com/PaddlePaddle/Paddle/go/utils/networkhelper"
	"github.com/coreos/etcd/clientv3"
	"github.com/coreos/etcd/clientv3/concurrency"
	log "github.com/sirupsen/logrus"
)

// ElementType is the type of elements of a Parameter.
type ElementType int

const (
	AlreadyInitialized = "pserver already initialized"
	Uninitialized      = "pserver not fully initialized"
)

// Supported element types
const (
	Int32 ElementType = iota
	UInt32
	Int64
	UInt64
	Float32
	Float64
)

// PsDesired is etcd path for store desired pserver count
const PsDesired = "/ps_desired"

// Parameter is a piece of data to sync with the parameter server.
type Parameter struct {
	Name        string
	ElementType ElementType
	Content     []byte
}

// ParameterWithConfig contains the parameter and the configuration.
type ParameterWithConfig struct {
	Param  Parameter
	Config []byte // parameter configuration in Proto Buffer format
}

// Gradient is the gradient of the parameter.
type Gradient Parameter

// Service is the RPC service for pserver.
type Service struct {
	initialized chan struct{}

	mu       sync.Mutex
	opt      *optimizer
	paramMap map[string]Parameter

	etcdEndpoints string
	etcdClient    *clientv3.Client
	// etcdTimeout is also used as retry intervals.
	etcdTimeout time.Duration
	// desired number of pservers in the job.
	// assume desired will not change during one training job.
	desired int
	// FIXME: ensure GetExternalIP gets the correct ip for trainers to connect.
	externalIP string
}

// NewService creates a new service, will bypass etcd registration if no
// endpoints specified.
func NewService(endpoints string, timeout time.Duration) (*Service, error) {
	s := &Service{opt: newOptimizer(sgd, 0.005)}
	s.paramMap = make(map[string]Parameter)
	s.initialized = make(chan struct{})
	s.etcdEndpoints = endpoints
	s.etcdTimeout = timeout

	var err error
	s.externalIP, err = networkhelper.GetExternalIP()
	if err != nil {
		return nil, err
	}

	if endpoints != "" {
		// initialize connection to etcd, try
		ep := strings.Split(s.etcdEndpoints, ",")
		for {
			cli, err := clientv3.New(clientv3.Config{
				Endpoints:   ep,
				DialTimeout: s.etcdTimeout,
			})
			if err != nil {
				log.Errorf("connect to etcd error: %v", err)
				time.Sleep(s.etcdTimeout)
				continue
			}
			s.etcdClient = cli
			log.Debugf("inited client to %s", s.etcdEndpoints)
			break
		}
		// wait and set s.desired init value
		for {
			ctx, cancel := context.WithTimeout(context.Background(), time.Second)
			resp, err := s.etcdClient.Get(ctx, PsDesired)
			cancel()
			if err != nil {
				log.Errorf("getting %s error: %v", PsDesired, err)
				time.Sleep(s.etcdTimeout)
				continue
			}
			if len(resp.Kvs) != 0 {
				s.desired, err = strconv.Atoi(string(resp.Kvs[0].Value))
				if err != nil {
					log.Errorf("value of %s invalid %v\n", PsDesired, err)
					time.Sleep(s.etcdTimeout)
					// NOTE: wait util ps_desired value change
					continue
				}
				break
			}
		}
		// try register pserver node on etcd
		for {
			ctx, cancel := context.WithTimeout(context.Background(), time.Second)
			_, err := s.registerPserverEtcd(ctx)
			cancel()
			if err != nil {
				log.Warn(err)
				time.Sleep(s.etcdTimeout)
				continue
			}
			break
		}
	} // if endpoints != ""
	// Bypass etcd registration if no endpoints specified
	return s, nil
}

// registerPserverEtcd registers pserver node on etcd using transaction.
func (s *Service) registerPserverEtcd(ctx context.Context) (*clientv3.TxnResponse, error) {
	return concurrency.NewSTM(s.etcdClient, func(c concurrency.STM) error {
		registered := false
		for i := 0; i < s.desired; i++ {
			psKey := "/ps/" + strconv.Itoa(i)
			log.Debugf("checking %s", psKey)
			ps := c.Get(psKey)
			log.Debugf("got value (%s) for key: %s", ps, psKey)

			if ps == "" {
				resp, err := s.etcdClient.Grant(context.TODO(), 5)
				if err != nil {
					log.Fatal(err)
				}
				// find the first id and write info
				c.Put(psKey, s.externalIP, clientv3.WithLease(resp.ID))
				log.Debugf("set pserver node %s with value %s", psKey, s.externalIP)
				_, kaerr := s.etcdClient.KeepAlive(context.TODO(), resp.ID)
				if kaerr != nil {
					log.Errorf("keepalive etcd node error: %v", kaerr)
					return kaerr
				}
				log.Debug("register finished")
				registered = true
				break
			}
		}
		if registered == true {
			return nil
		}
		return errors.New("not registerd, may due to already have enough pservers")
	}, concurrency.WithAbortContext(ctx), concurrency.WithIsolation(concurrency.RepeatableReads))
}

// InitParam initializes a parameter.
func (s *Service) InitParam(paramWithConfigs ParameterWithConfig, dummy *int) error {
	select {
	case <-s.initialized:
		return errors.New(AlreadyInitialized)
	default:
	}

	// TODO(helin): parse parameter config

	s.mu.Lock()
	defer s.mu.Unlock()

	// TODO(helin): check if paramWithConfigs.Param.Content is
	// properly memory aligned, if not, make copy to a memory
	// aligned region.
	s.paramMap[paramWithConfigs.Param.Name] = paramWithConfigs.Param
	return nil
}

// FinishInitParams tells the parameter server that the parameter
// initialization has finished.
func (s *Service) FinishInitParams(dummy0 int, dummy1 *int) error {
	select {
	case <-s.initialized:
		return errors.New(AlreadyInitialized)
	default:
	}

	close(s.initialized)
	return nil
}

// SendGrad sends gradient to parameter servers for parameter
// optimization.
func (s *Service) SendGrad(g Gradient, dummy *int) error {
	select {
	case <-s.initialized:
	default:
		return errors.New(Uninitialized)
	}

	s.mu.Lock()
	defer s.mu.Unlock()

	p, ok := s.paramMap[g.Name]
	if !ok {
		return fmt.Errorf("parameter: %s does not exist", g.Name)
	}

	return s.opt.UpdateParameter(p, g)
}

// GetParam gets parameters from the parameter server.
func (s *Service) GetParam(name string, parameter *Parameter) error {
	<-s.initialized
	s.mu.Lock()
	defer s.mu.Unlock()

	p, ok := s.paramMap[name]
	if !ok {
		return fmt.Errorf("parameter: %s does not exist", name)
	}

	// The parameter content (a byte slice) may change
	// during RPC serialization due to write from other
	// goroutine, we allow it since mini-batch based deep
	// learning optimization methods are stochastic in
	// nature. This race condition is allowed deliberately
	// to save the program from making a copy of the
	// paramter content.
	*parameter = p
	return nil
}

// Save tells the parameter server to save parameters.
func (s *Service) Save(path string, dummy *int) error {
	<-s.initialized

	// TODO
	return nil
}
<commit_msg>fix etcd lease<commit_after>package pserver

import (
	"context"
	"errors"
	"fmt"
	"strconv"
	"strings"
	"sync"
	"time"

	"github.com/PaddlePaddle/Paddle/go/utils/networkhelper"
	"github.com/coreos/etcd/clientv3"
	"github.com/coreos/etcd/clientv3/concurrency"
	log "github.com/sirupsen/logrus"
)

// ElementType is the type of elements of a Parameter.
type ElementType int

const (
	AlreadyInitialized = "pserver already initialized"
	Uninitialized      = "pserver not fully initialized"
)

// Supported element types
const (
	Int32 ElementType = iota
	UInt32
	Int64
	UInt64
	Float32
	Float64
)

// PsDesired is etcd path for store desired pserver count
const PsDesired = "/ps_desired"

// Parameter is a piece of data to sync with the parameter server.
type Parameter struct {
	Name        string
	ElementType ElementType
	Content     []byte
}

// ParameterWithConfig contains the parameter and the configuration.
type ParameterWithConfig struct {
	Param  Parameter
	Config []byte // parameter configuration in Proto Buffer format
}

// Gradient is the gradient of the parameter.
type Gradient Parameter

// Service is the RPC service for pserver.
type Service struct {
	initialized chan struct{}

	mu       sync.Mutex
	opt      *optimizer
	paramMap map[string]Parameter

	etcdEndpoints string
	etcdClient    *clientv3.Client
	// etcdTimeout is also used as retry intervals.
	etcdTimeout time.Duration
	// desired number of pservers in the job.
	// assume desired will not change during one training job.
	desired int
	// FIXME: ensure GetExternalIP gets the correct ip for trainers to connect.
	externalIP string
}

// NewService creates a new service, will bypass etcd registration if no
// endpoints specified.
func NewService(endpoints string, timeout time.Duration) (*Service, error) {
	s := &Service{opt: newOptimizer(sgd, 0.005)}
	s.paramMap = make(map[string]Parameter)
	s.initialized = make(chan struct{})
	s.etcdEndpoints = endpoints
	s.etcdTimeout = timeout

	var err error
	s.externalIP, err = networkhelper.GetExternalIP()
	if err != nil {
		return nil, err
	}

	if endpoints != "" {
		// initialize connection to etcd, try
		ep := strings.Split(s.etcdEndpoints, ",")
		for {
			cli, err := clientv3.New(clientv3.Config{
				Endpoints:   ep,
				DialTimeout: s.etcdTimeout,
			})
			if err != nil {
				log.Errorf("connect to etcd error: %v", err)
				time.Sleep(s.etcdTimeout)
				continue
			}
			s.etcdClient = cli
			log.Debugf("inited client to %s", s.etcdEndpoints)
			break
		}
		// wait and set s.desired init value
		for {
			ctx, cancel := context.WithTimeout(context.Background(), time.Second)
			resp, err := s.etcdClient.Get(ctx, PsDesired)
			cancel()
			if err != nil {
				log.Errorf("getting %s error: %v", PsDesired, err)
				time.Sleep(s.etcdTimeout)
				continue
			}
			if len(resp.Kvs) != 0 {
				s.desired, err = strconv.Atoi(string(resp.Kvs[0].Value))
				if err != nil {
					log.Errorf("value of %s invalid %v\n", PsDesired, err)
					time.Sleep(s.etcdTimeout)
					// NOTE: wait util ps_desired value change
					continue
				}
				break
			}
		}
		// try register pserver node on etcd
		for {
			ctx, cancel := context.WithTimeout(context.Background(), time.Second)
			_, err := s.registerPserverEtcd(ctx)
			cancel()
			if err != nil {
				log.Warn(err)
				time.Sleep(s.etcdTimeout)
				continue
			}
			break
		}
	} // if endpoints != ""
	// Bypass etcd registration if no endpoints specified
	return s, nil
}

// registerPserverEtcd registers pserver node on etcd using transaction.
func (s *Service) registerPserverEtcd(ctx context.Context) (*clientv3.TxnResponse, error) {
	return concurrency.NewSTM(s.etcdClient, func(c concurrency.STM) error {
		registered := false
		for i := 0; i < s.desired; i++ {
			psKey := "/ps/" + strconv.Itoa(i)
			log.Debugf("checking %s", psKey)
			ps := c.Get(psKey)
			log.Debugf("got value (%s) for key: %s", ps, psKey)

			if ps == "" {
				resp, err := s.etcdClient.Grant(context.TODO(), 5)
				if err != nil {
					log.Fatal(err)
				}
				// find the first id and write info
				c.Put(psKey, s.externalIP, clientv3.WithLease(resp.ID))
				log.Debugf("set pserver node %s with value %s", psKey, s.externalIP)
				ch, kaerr := s.etcdClient.KeepAlive(context.TODO(), resp.ID)
				if kaerr != nil {
					log.Errorf("keepalive etcd node error: %v", kaerr)
					return kaerr
				}

				// Eat the keep alive message so etcd
				// will not expire the lease.
				go func(ch <-chan *clientv3.LeaseKeepAliveResponse) {
					ka := <-ch
					log.Debugf("keepalive: %d\n", ka.TTL)
				}(ch)
				log.Debug("register finished")
				registered = true
				break
			}
		}
		if registered == true {
			return nil
		}
		return errors.New("not registerd, may due to already have enough pservers")
	}, concurrency.WithAbortContext(ctx), concurrency.WithIsolation(concurrency.RepeatableReads))
}

// InitParam initializes a parameter.
func (s *Service) InitParam(paramWithConfigs ParameterWithConfig, dummy *int) error {
	select {
	case <-s.initialized:
		return errors.New(AlreadyInitialized)
	default:
	}

	// TODO(helin): parse parameter config

	s.mu.Lock()
	defer s.mu.Unlock()

	// TODO(helin): check if paramWithConfigs.Param.Content is
	// properly memory aligned, if not, make copy to a memory
	// aligned region.
	s.paramMap[paramWithConfigs.Param.Name] = paramWithConfigs.Param
	return nil
}

// FinishInitParams tells the parameter server that the parameter
// initialization has finished.
func (s *Service) FinishInitParams(dummy0 int, dummy1 *int) error {
	select {
	case <-s.initialized:
		return errors.New(AlreadyInitialized)
	default:
	}

	close(s.initialized)
	return nil
}

// SendGrad sends gradient to parameter servers for parameter
// optimization.
func (s *Service) SendGrad(g Gradient, dummy *int) error {
	select {
	case <-s.initialized:
	default:
		return errors.New(Uninitialized)
	}

	s.mu.Lock()
	defer s.mu.Unlock()

	p, ok := s.paramMap[g.Name]
	if !ok {
		return fmt.Errorf("parameter: %s does not exist", g.Name)
	}

	return s.opt.UpdateParameter(p, g)
}

// GetParam gets parameters from the parameter server.
func (s *Service) GetParam(name string, parameter *Parameter) error {
	<-s.initialized
	s.mu.Lock()
	defer s.mu.Unlock()

	p, ok := s.paramMap[name]
	if !ok {
		return fmt.Errorf("parameter: %s does not exist", name)
	}

	// The parameter content (a byte slice) may change
	// during RPC serialization due to write from other
	// goroutine, we allow it since mini-batch based deep
	// learning optimization methods are stochastic in
	// nature. This race condition is allowed deliberately
	// to save the program from making a copy of the
	// paramter content.
	*parameter = p
	return nil
}

// Save tells the parameter server to save parameters.
func (s *Service) Save(path string, dummy *int) error {
	<-s.initialized

	// TODO
	return nil
}
<|endoftext|>


<commit_before>/*
Copyright 2019 The Vitess Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package vtctl

import (
	"errors"
	"flag"
	"fmt"
	"io"
	"time"

	"context"

	"vitess.io/vitess/go/vt/logutil"
	"vitess.io/vitess/go/vt/mysqlctl"
	"vitess.io/vitess/go/vt/mysqlctl/backupstorage"
	topodatapb "vitess.io/vitess/go/vt/proto/topodata"
	vtrpcpb "vitess.io/vitess/go/vt/proto/vtrpc"
	"vitess.io/vitess/go/vt/topo/topoproto"
	"vitess.io/vitess/go/vt/vterrors"
	"vitess.io/vitess/go/vt/wrangler"
)

func init() {
	addCommand("Shards", command{
		"ListBackups",
		commandListBackups,
		"<keyspace/shard>",
		"Lists all the backups for a shard."})
	addCommand("Shards", command{
		"BackupShard",
		commandBackupShard,
		"[-allow_primary=false] <keyspace/shard>",
		"Chooses a tablet and creates a backup for a shard."})
	addCommand("Shards", command{
		"RemoveBackup",
		commandRemoveBackup,
		"<keyspace/shard> <backup name>",
		"Removes a backup for the BackupStorage."})

	addCommand("Tablets", command{
		"Backup",
		commandBackup,
		"[-concurrency=4] [-allow_primary=false] <tablet alias>",
		"Stops mysqld and uses the BackupStorage service to store a new backup. This function also remembers if the tablet was replicating so that it can restore the same state after the backup completes."})
	addCommand("Tablets", command{
		"RestoreFromBackup",
		commandRestoreFromBackup,
		"[-backup_timestamp=yyyy-MM-dd.HHmmss] <tablet alias>",
		"Stops mysqld and restores the data from the latest backup or if a timestamp is specified then the most recent backup at or before that time."})
}

func commandBackup(ctx context.Context, wr *wrangler.Wrangler, subFlags *flag.FlagSet, args []string) error {
	concurrency := subFlags.Int("concurrency", 4, "Specifies the number of compression/checksum jobs to run simultaneously")
	allowPrimary := subFlags.Bool("allow_primary", false, "Allows backups to be taken on primary. Warning!! If you are using the builtin backup engine, this will shutdown your primary mysql for as long as it takes to create a backup.")

	// handle deprecated flags
	// should be deleted in a future release
	deprecatedAllowMaster := subFlags.Bool("allow_master", false, "DEPRECATED. Use -allow_primary instead")

	if err := subFlags.Parse(args); err != nil {
		return err
	}
	if subFlags.NArg() != 1 {
		return fmt.Errorf("the Backup command requires the <tablet alias> argument")
	}

	tabletAlias, err := topoproto.ParseTabletAlias(subFlags.Arg(0))
	if err != nil {
		return err
	}
	tabletInfo, err := wr.TopoServer().GetTablet(ctx, tabletAlias)
	if err != nil {
		return err
	}
	if *deprecatedAllowMaster {
		*allowPrimary = *deprecatedAllowMaster
	}

	return execBackup(ctx, wr, tabletInfo.Tablet, *concurrency, *allowPrimary)
}

func commandBackupShard(ctx context.Context, wr *wrangler.Wrangler, subFlags *flag.FlagSet, args []string) error {
	concurrency := subFlags.Int("concurrency", 4, "Specifies the number of compression/checksum jobs to run simultaneously")
	allowPrimary := subFlags.Bool("allow_primary", false, "Whether to use primary tablet for backup. Warning!! If you are using the builtin backup engine, this will shutdown your primary mysql for as long as it takes to create a backup.")

	// handle deprecated flags
	// should be deleted in a future release
	deprecatedAllowMaster := subFlags.Bool("allow_master", false, "DEPRECATED. Use -allow_primary instead")

	if err := subFlags.Parse(args); err != nil {
		return err
	}
	if subFlags.NArg() != 1 {
		return fmt.Errorf("action BackupShard requires <keyspace/shard>")
	}

	keyspace, shard, err := topoproto.ParseKeyspaceShard(subFlags.Arg(0))
	if err != nil {
		return err
	}

	tablets, stats, err := wr.ShardReplicationStatuses(ctx, keyspace, shard)
	if tablets == nil {
		return err
	}

	var tabletForBackup *topodatapb.Tablet
	var secondsBehind uint32

	for i := range tablets {
		// find a replica, rdonly or spare tablet type to run the backup on
		switch tablets[i].Type {
		case topodatapb.TabletType_REPLICA, topodatapb.TabletType_RDONLY, topodatapb.TabletType_SPARE:
		default:
			continue
		}
		// choose the first tablet as the baseline
		if tabletForBackup == nil {
			tabletForBackup = tablets[i].Tablet
			secondsBehind = stats[i].ReplicationLagSeconds
			continue
		}

		// choose a new tablet if it is more up to date
		if stats[i].ReplicationLagSeconds < secondsBehind {
			tabletForBackup = tablets[i].Tablet
			secondsBehind = stats[i].ReplicationLagSeconds
		}
	}

	// if no other tablet is available and allowPrimary is set to true
	if tabletForBackup == nil && *allowPrimary {
	ChooseTablet:
		for i := range tablets {
			switch tablets[i].Type {
			case topodatapb.TabletType_PRIMARY:
				tabletForBackup = tablets[i].Tablet
				secondsBehind = 0 //nolint
				break ChooseTablet
			default:
				continue
			}
		}
	}

	if tabletForBackup == nil {
		return errors.New("no tablet available for backup")
	}

	if *deprecatedAllowMaster {
		*allowPrimary = *deprecatedAllowMaster
	}

	return execBackup(ctx, wr, tabletForBackup, *concurrency, *allowPrimary)
}

// execBackup is shared by Backup and BackupShard
func execBackup(ctx context.Context, wr *wrangler.Wrangler, tablet *topodatapb.Tablet, concurrency int, allowPrimary bool) error {
	stream, err := wr.TabletManagerClient().Backup(ctx, tablet, concurrency, allowPrimary)
	if err != nil {
		return err
	}
	for {
		e, err := stream.Recv()
		switch err {
		case nil:
			logutil.LogEvent(wr.Logger(), e)
		case io.EOF:
			return nil
		default:
			return err
		}
	}
}

func commandListBackups(ctx context.Context, wr *wrangler.Wrangler, subFlags *flag.FlagSet, args []string) error {
	if err := subFlags.Parse(args); err != nil {
		return err
	}
	if subFlags.NArg() != 1 {
		return fmt.Errorf("action ListBackups requires <keyspace/shard>")
	}

	keyspace, shard, err := topoproto.ParseKeyspaceShard(subFlags.Arg(0))
	if err != nil {
		return err
	}
	bucket := fmt.Sprintf("%v/%v", keyspace, shard)

	bs, err := backupstorage.GetBackupStorage()
	if err != nil {
		return err
	}
	defer bs.Close()
	bhs, err := bs.ListBackups(ctx, bucket)
	if err != nil {
		return err
	}
	for _, bh := range bhs {
		wr.Logger().Printf("%v\n", bh.Name())
	}
	return nil
}

func commandRemoveBackup(ctx context.Context, wr *wrangler.Wrangler, subFlags *flag.FlagSet, args []string) error {
	if err := subFlags.Parse(args); err != nil {
		return err
	}
	if subFlags.NArg() != 2 {
		return fmt.Errorf("action RemoveBackup requires <keyspace/shard> <backup name>")
	}

	keyspace, shard, err := topoproto.ParseKeyspaceShard(subFlags.Arg(0))
	if err != nil {
		return err
	}
	bucket := fmt.Sprintf("%v/%v", keyspace, shard)
	name := subFlags.Arg(1)

	bs, err := backupstorage.GetBackupStorage()
	if err != nil {
		return err
	}
	defer bs.Close()
	return bs.RemoveBackup(ctx, bucket, name)
}

func commandRestoreFromBackup(ctx context.Context, wr *wrangler.Wrangler, subFlags *flag.FlagSet, args []string) error {
	backupTimestampStr := subFlags.String("backup_timestamp", "", "Use the backup taken at or before this timestamp rather than using the latest backup.")
	if err := subFlags.Parse(args); err != nil {
		return err
	}
	if subFlags.NArg() != 1 {
		return fmt.Errorf("the RestoreFromBackup command requires the <tablet alias> argument")
	}

	// Zero date will cause us to use the latest, which is the default
	backupTime := time.Time{}

	// Or if a backup timestamp was specified then we use the last backup taken at or before that time
	if *backupTimestampStr != "" {
		var err error
		backupTime, err = time.Parse(mysqlctl.BackupTimestampFormat, *backupTimestampStr)
		if err != nil {
			return vterrors.New(vtrpcpb.Code_INVALID_ARGUMENT, fmt.Sprintf("unable to parse the backup timestamp value provided of '%s'", *backupTimestampStr))
		}
	}

	tabletAlias, err := topoproto.ParseTabletAlias(subFlags.Arg(0))
	if err != nil {
		return err
	}
	tabletInfo, err := wr.TopoServer().GetTablet(ctx, tabletAlias)
	if err != nil {
		return err
	}
	stream, err := wr.TabletManagerClient().RestoreFromBackup(ctx, tabletInfo.Tablet, backupTime)
	if err != nil {
		return err
	}
	for {
		e, err := stream.Recv()
		switch err {
		case nil:
			logutil.LogEvent(wr.Logger(), e)
		case io.EOF:
			return nil
		default:
			return err
		}
	}
}
<commit_msg>vtctl: check ShardReplicationStatuses error<commit_after>/*
Copyright 2019 The Vitess Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package vtctl

import (
	"context"
	"errors"
	"flag"
	"fmt"
	"io"
	"time"

	"vitess.io/vitess/go/vt/logutil"
	"vitess.io/vitess/go/vt/mysqlctl"
	"vitess.io/vitess/go/vt/mysqlctl/backupstorage"
	topodatapb "vitess.io/vitess/go/vt/proto/topodata"
	vtrpcpb "vitess.io/vitess/go/vt/proto/vtrpc"
	"vitess.io/vitess/go/vt/topo/topoproto"
	"vitess.io/vitess/go/vt/vterrors"
	"vitess.io/vitess/go/vt/wrangler"
)

func init() {
	addCommand("Shards", command{
		"ListBackups",
		commandListBackups,
		"<keyspace/shard>",
		"Lists all the backups for a shard."})
	addCommand("Shards", command{
		"BackupShard",
		commandBackupShard,
		"[-allow_primary=false] <keyspace/shard>",
		"Chooses a tablet and creates a backup for a shard."})
	addCommand("Shards", command{
		"RemoveBackup",
		commandRemoveBackup,
		"<keyspace/shard> <backup name>",
		"Removes a backup for the BackupStorage."})

	addCommand("Tablets", command{
		"Backup",
		commandBackup,
		"[-concurrency=4] [-allow_primary=false] <tablet alias>",
		"Stops mysqld and uses the BackupStorage service to store a new backup. This function also remembers if the tablet was replicating so that it can restore the same state after the backup completes."})
	addCommand("Tablets", command{
		"RestoreFromBackup",
		commandRestoreFromBackup,
		"[-backup_timestamp=yyyy-MM-dd.HHmmss] <tablet alias>",
		"Stops mysqld and restores the data from the latest backup or if a timestamp is specified then the most recent backup at or before that time."})
}

func commandBackup(ctx context.Context, wr *wrangler.Wrangler, subFlags *flag.FlagSet, args []string) error {
	concurrency := subFlags.Int("concurrency", 4, "Specifies the number of compression/checksum jobs to run simultaneously")
	allowPrimary := subFlags.Bool("allow_primary", false, "Allows backups to be taken on primary. Warning!! If you are using the builtin backup engine, this will shutdown your primary mysql for as long as it takes to create a backup.")

	// handle deprecated flags
	// should be deleted in a future release
	deprecatedAllowMaster := subFlags.Bool("allow_master", false, "DEPRECATED. Use -allow_primary instead")

	if err := subFlags.Parse(args); err != nil {
		return err
	}
	if subFlags.NArg() != 1 {
		return fmt.Errorf("the Backup command requires the <tablet alias> argument")
	}

	tabletAlias, err := topoproto.ParseTabletAlias(subFlags.Arg(0))
	if err != nil {
		return err
	}
	tabletInfo, err := wr.TopoServer().GetTablet(ctx, tabletAlias)
	if err != nil {
		return err
	}
	if *deprecatedAllowMaster {
		*allowPrimary = *deprecatedAllowMaster
	}

	return execBackup(ctx, wr, tabletInfo.Tablet, *concurrency, *allowPrimary)
}

func commandBackupShard(ctx context.Context, wr *wrangler.Wrangler, subFlags *flag.FlagSet, args []string) error {
	concurrency := subFlags.Int("concurrency", 4, "Specifies the number of compression/checksum jobs to run simultaneously")
	allowPrimary := subFlags.Bool("allow_primary", false, "Whether to use primary tablet for backup. Warning!! If you are using the builtin backup engine, this will shutdown your primary mysql for as long as it takes to create a backup.")

	// handle deprecated flags
	// should be deleted in a future release
	deprecatedAllowMaster := subFlags.Bool("allow_master", false, "DEPRECATED. Use -allow_primary instead")

	if err := subFlags.Parse(args); err != nil {
		return err
	}
	if subFlags.NArg() != 1 {
		return fmt.Errorf("action BackupShard requires <keyspace/shard>")
	}

	keyspace, shard, err := topoproto.ParseKeyspaceShard(subFlags.Arg(0))
	if err != nil {
		return err
	}

	tablets, stats, err := wr.ShardReplicationStatuses(ctx, keyspace, shard)
	if err != nil {
		return err
	}

	var tabletForBackup *topodatapb.Tablet
	var secondsBehind uint32

	for i := range tablets {
		// find a replica, rdonly or spare tablet type to run the backup on
		switch tablets[i].Type {
		case topodatapb.TabletType_REPLICA, topodatapb.TabletType_RDONLY, topodatapb.TabletType_SPARE:
		default:
			continue
		}
		// choose the first tablet as the baseline
		if tabletForBackup == nil {
			tabletForBackup = tablets[i].Tablet
			secondsBehind = stats[i].ReplicationLagSeconds
			continue
		}

		// choose a new tablet if it is more up to date
		if stats[i].ReplicationLagSeconds < secondsBehind {
			tabletForBackup = tablets[i].Tablet
			secondsBehind = stats[i].ReplicationLagSeconds
		}
	}

	// if no other tablet is available and allowPrimary is set to true
	if tabletForBackup == nil && *allowPrimary {
	ChooseTablet:
		for i := range tablets {
			switch tablets[i].Type {
			case topodatapb.TabletType_PRIMARY:
				tabletForBackup = tablets[i].Tablet
				secondsBehind = 0 //nolint
				break ChooseTablet
			default:
				continue
			}
		}
	}

	if tabletForBackup == nil {
		return errors.New("no tablet available for backup")
	}

	if *deprecatedAllowMaster {
		*allowPrimary = *deprecatedAllowMaster
	}

	return execBackup(ctx, wr, tabletForBackup, *concurrency, *allowPrimary)
}

// execBackup is shared by Backup and BackupShard
func execBackup(ctx context.Context, wr *wrangler.Wrangler, tablet *topodatapb.Tablet, concurrency int, allowPrimary bool) error {
	stream, err := wr.TabletManagerClient().Backup(ctx, tablet, concurrency, allowPrimary)
	if err != nil {
		return err
	}
	for {
		e, err := stream.Recv()
		switch err {
		case nil:
			logutil.LogEvent(wr.Logger(), e)
		case io.EOF:
			return nil
		default:
			return err
		}
	}
}

func commandListBackups(ctx context.Context, wr *wrangler.Wrangler, subFlags *flag.FlagSet, args []string) error {
	if err := subFlags.Parse(args); err != nil {
		return err
	}
	if subFlags.NArg() != 1 {
		return fmt.Errorf("action ListBackups requires <keyspace/shard>")
	}

	keyspace, shard, err := topoproto.ParseKeyspaceShard(subFlags.Arg(0))
	if err != nil {
		return err
	}
	bucket := fmt.Sprintf("%v/%v", keyspace, shard)

	bs, err := backupstorage.GetBackupStorage()
	if err != nil {
		return err
	}
	defer bs.Close()
	bhs, err := bs.ListBackups(ctx, bucket)
	if err != nil {
		return err
	}
	for _, bh := range bhs {
		wr.Logger().Printf("%v\n", bh.Name())
	}
	return nil
}

func commandRemoveBackup(ctx context.Context, wr *wrangler.Wrangler, subFlags *flag.FlagSet, args []string) error {
	if err := subFlags.Parse(args); err != nil {
		return err
	}
	if subFlags.NArg() != 2 {
		return fmt.Errorf("action RemoveBackup requires <keyspace/shard> <backup name>")
	}

	keyspace, shard, err := topoproto.ParseKeyspaceShard(subFlags.Arg(0))
	if err != nil {
		return err
	}
	bucket := fmt.Sprintf("%v/%v", keyspace, shard)
	name := subFlags.Arg(1)

	bs, err := backupstorage.GetBackupStorage()
	if err != nil {
		return err
	}
	defer bs.Close()
	return bs.RemoveBackup(ctx, bucket, name)
}

func commandRestoreFromBackup(ctx context.Context, wr *wrangler.Wrangler, subFlags *flag.FlagSet, args []string) error {
	backupTimestampStr := subFlags.String("backup_timestamp", "", "Use the backup taken at or before this timestamp rather than using the latest backup.")
	if err := subFlags.Parse(args); err != nil {
		return err
	}
	if subFlags.NArg() != 1 {
		return fmt.Errorf("the RestoreFromBackup command requires the <tablet alias> argument")
	}

	// Zero date will cause us to use the latest, which is the default
	backupTime := time.Time{}

	// Or if a backup timestamp was specified then we use the last backup taken at or before that time
	if *backupTimestampStr != "" {
		var err error
		backupTime, err = time.Parse(mysqlctl.BackupTimestampFormat, *backupTimestampStr)
		if err != nil {
			return vterrors.New(vtrpcpb.Code_INVALID_ARGUMENT, fmt.Sprintf("unable to parse the backup timestamp value provided of '%s'", *backupTimestampStr))
		}
	}

	tabletAlias, err := topoproto.ParseTabletAlias(subFlags.Arg(0))
	if err != nil {
		return err
	}
	tabletInfo, err := wr.TopoServer().GetTablet(ctx, tabletAlias)
	if err != nil {
		return err
	}
	stream, err := wr.TabletManagerClient().RestoreFromBackup(ctx, tabletInfo.Tablet, backupTime)
	if err != nil {
		return err
	}
	for {
		e, err := stream.Recv()
		switch err {
		case nil:
			logutil.LogEvent(wr.Logger(), e)
		case io.EOF:
			return nil
		default:
			return err
		}
	}
}
<|endoftext|>


<commit_before>package gqt_test

import (
	"os"
	"runtime"

	"github.com/cloudfoundry-incubator/guardian/gqt/runner"
	. "github.com/onsi/ginkgo"
	. "github.com/onsi/gomega"
	"github.com/onsi/gomega/gexec"

	"testing"
)

var OciRuntimeBin = os.Getenv("OCI_RUNTIME")

var defaultRuntime = map[string]string{
	"linux":  "runc",
	"darwin": "nocs",
}

func TestGqt(t *testing.T) {
	RegisterFailHandler(Fail)

	if OciRuntimeBin == "" {
		OciRuntimeBin = defaultRuntime[runtime.GOOS]
	}

	RunSpecs(t, "Gqt Suite")
}

func startGarden(argv ...string) *runner.RunningGarden {
	gardenBin, err := gexec.Build("github.com/cloudfoundry-incubator/guardian/cmd/guardian")
	Expect(err).NotTo(HaveOccurred())

	return runner.Start(gardenBin, argv...)
}
<commit_msg>Avoid running gqt on darwin<commit_after>package gqt_test

import (
	"os"
	"runtime"

	"github.com/cloudfoundry-incubator/guardian/gqt/runner"
	. "github.com/onsi/ginkgo"
	. "github.com/onsi/gomega"
	"github.com/onsi/gomega/gexec"

	"testing"
)

var OciRuntimeBin = os.Getenv("OCI_RUNTIME")

var defaultRuntime = map[string]string{
	"linux": "runc",
}

func TestGqt(t *testing.T) {
	RegisterFailHandler(Fail)

	BeforeEach(func() {
		if OciRuntimeBin == "" {
			OciRuntimeBin = defaultRuntime[runtime.GOOS]
		}

		if OciRuntimeBin == "" {
			Skip("No OCI Runtime for Platform: " + runtime.GOOS)
		}
	})

	RunSpecs(t, "Gqt Suite")
}

func startGarden(argv ...string) *runner.RunningGarden {
	gardenBin, err := gexec.Build("github.com/cloudfoundry-incubator/guardian/cmd/guardian")
	Expect(err).NotTo(HaveOccurred())

	return runner.Start(gardenBin, argv...)
}
<|endoftext|>


<commit_before>// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package gc

// Transmogrify slow integer division into fast multiplication using magic.

// argument passing to/from
// smagic and umagic
type Magic struct {
	W   int // input for both - width
	S   int // output for both - shift
	Bad int // output for both - unexpected failure

	// magic multiplier for signed literal divisors
	Sd int64 // input - literal divisor
	Sm int64 // output - multiplier

	// magic multiplier for unsigned literal divisors
	Ud uint64 // input - literal divisor
	Um uint64 // output - multiplier
	Ua int    // output - adder
}

// magic number for signed division
// see hacker's delight chapter 10
func smagic(m *Magic) {
	var mask uint64

	m.Bad = 0
	switch m.W {
	default:
		m.Bad = 1
		return

	case 8:
		mask = 0xff

	case 16:
		mask = 0xffff

	case 32:
		mask = 0xffffffff

	case 64:
		mask = 0xffffffffffffffff
	}

	two31 := mask ^ (mask >> 1)

	p := m.W - 1
	ad := uint64(m.Sd)
	if m.Sd < 0 {
		ad = -uint64(m.Sd)
	}

	// bad denominators
	if ad == 0 || ad == 1 || ad == two31 {
		m.Bad = 1
		return
	}

	t := two31
	ad &= mask

	anc := t - 1 - t%ad
	anc &= mask

	q1 := two31 / anc
	r1 := two31 - q1*anc
	q1 &= mask
	r1 &= mask

	q2 := two31 / ad
	r2 := two31 - q2*ad
	q2 &= mask
	r2 &= mask

	var delta uint64
	for {
		p++
		q1 <<= 1
		r1 <<= 1
		q1 &= mask
		r1 &= mask
		if r1 >= anc {
			q1++
			r1 -= anc
			q1 &= mask
			r1 &= mask
		}

		q2 <<= 1
		r2 <<= 1
		q2 &= mask
		r2 &= mask
		if r2 >= ad {
			q2++
			r2 -= ad
			q2 &= mask
			r2 &= mask
		}

		delta = ad - r2
		delta &= mask
		if q1 < delta || (q1 == delta && r1 == 0) {
			continue
		}

		break
	}

	m.Sm = int64(q2 + 1)
	if uint64(m.Sm)&two31 != 0 {
		m.Sm |= ^int64(mask)
	}
	m.S = p - m.W
}

// magic number for unsigned division
// see hacker's delight chapter 10
func umagic(m *Magic) {
	var mask uint64

	m.Bad = 0
	m.Ua = 0

	switch m.W {
	default:
		m.Bad = 1
		return

	case 8:
		mask = 0xff

	case 16:
		mask = 0xffff

	case 32:
		mask = 0xffffffff

	case 64:
		mask = 0xffffffffffffffff
	}

	two31 := mask ^ (mask >> 1)

	m.Ud &= mask
	if m.Ud == 0 || m.Ud == two31 {
		m.Bad = 1
		return
	}

	nc := mask - (-m.Ud&mask)%m.Ud
	p := m.W - 1

	q1 := two31 / nc
	r1 := two31 - q1*nc
	q1 &= mask
	r1 &= mask

	q2 := (two31 - 1) / m.Ud
	r2 := (two31 - 1) - q2*m.Ud
	q2 &= mask
	r2 &= mask

	var delta uint64
	for {
		p++
		if r1 >= nc-r1 {
			q1 <<= 1
			q1++
			r1 <<= 1
			r1 -= nc
		} else {
			q1 <<= 1
			r1 <<= 1
		}

		q1 &= mask
		r1 &= mask
		if r2+1 >= m.Ud-r2 {
			if q2 >= two31-1 {
				m.Ua = 1
			}

			q2 <<= 1
			q2++
			r2 <<= 1
			r2++
			r2 -= m.Ud
		} else {
			if q2 >= two31 {
				m.Ua = 1
			}

			q2 <<= 1
			r2 <<= 1
			r2++
		}

		q2 &= mask
		r2 &= mask

		delta = m.Ud - 1 - r2
		delta &= mask

		if p < m.W+m.W {
			if q1 < delta || (q1 == delta && r1 == 0) {
				continue
			}
		}

		break
	}

	m.Um = q2 + 1
	m.S = p - m.W
}
<commit_msg>cmd/compile: remove unused constant divide strength reduction code<commit_after><|endoftext|>


<commit_before>package haaasd

import (
	"bytes"
	"errors"
	"fmt"
	log "github.com/Sirupsen/logrus"
	"io/ioutil"
	"os"
	"os/exec"
	"time"
)

func NewHaproxy(properties *Config, application string, platform string, version string) *Haproxy {
	if version == "" {
		version = "1.4.22"
	}
	return &Haproxy{
		Application: application,
		Platform:    platform,
		properties:  properties,
		Version:     version,
	}
}

type Haproxy struct {
	Application string
	Platform    string
	Version     string
	properties  *Config
	State       int
}

const (
	SUCCESS int = iota
	UNCHANGED int = iota
	ERR_SYSLOG int = iota
	ERR_CONF int = iota
	ERR_RELOAD int = iota
)

// ApplyConfiguration write the new configuration and reload
// A rollback is called on failure
func (hap *Haproxy) ApplyConfiguration(data *EventMessage) (int, error) {
	hap.createSkeleton(data.Correlationid)

	newConf := data.Conf
	path := hap.confPath()

	// Check conf diff
	oldConf, err := ioutil.ReadFile(path)

	if err != nil {
		log.WithFields(log.Fields{
			"correlationId" : data.Correlationid,
			"path": path,
		}).Error("Cannot read old configuration")
		return ERR_CONF, err
	}
	if log.GetLevel() == log.DebugLevel {
		hap.dumpConfiguration(hap.NewDebugPath(), newConf, data)
	}
	if bytes.Equal(oldConf, newConf) {
		log.WithFields(log.Fields{
			"correlationId": data.Correlationid,
			"application": data.Application,
			"plateform":   data.Platform,
		}).Info("Ignore unchanged configuration")
		return UNCHANGED, nil
	}

	// Archive previous configuration
	archivePath := hap.confArchivePath()
	os.Rename(path, archivePath)
	log.WithFields(
		log.Fields{
			"correlationId": data.Correlationid,
			"application": data.Application,
			"plateform":   data.Platform,
			"archivePath": archivePath,
		}).Info("Old configuration saved")
	err = ioutil.WriteFile(path, newConf, 0644)
	if err != nil {
		return ERR_CONF, err
	}
	log.WithField("path", path).Info("New configuration written")

	// Reload haproxy
	err = hap.reload(data.Correlationid)
	if err != nil {
		log.WithFields(log.Fields{
			"application": data.Application,
			"plateform":   data.Platform,
		}).WithError(err).Error("Reload failed")
		hap.dumpConfiguration(hap.NewErrorPath(), newConf, data)
		err = hap.rollback(data.Correlationid)
		return ERR_RELOAD, err
	}
	// Write syslog fragment
	fragmentPath := hap.syslogFragmentPath()
	err = ioutil.WriteFile(fragmentPath, data.SyslogFragment, 0644)
	if err != nil {
		log.WithFields(log.Fields{
			"application": data.Application,
			"plateform":   data.Platform,
		}).WithError(err).Error("Failed to write syslog fragment")
		// TODO Should we rollback on syslog error ?
		return ERR_SYSLOG, err
	}

	return SUCCESS, nil
}

// dumpConfiguration dumps the new configuration file with context for debugging purpose
func (hap *Haproxy) dumpConfiguration(filename string, newConf []byte, data *EventMessage) {

	f, err2 := os.Create(filename)
	defer f.Close()
	if err2 == nil {
		f.WriteString("================================================================\n")
		f.WriteString(fmt.Sprintf("application: %s\n", data.Application))
		f.WriteString(fmt.Sprintf("platform: %s\n", data.Platform))
		f.WriteString(fmt.Sprintf("correlationid: %s\n", data.Correlationid))
		f.WriteString("================================================================\n")
		f.Write(newConf)
		f.Sync()

		log.WithFields(log.Fields{
			"correlationId": data.Correlationid,
			"filename": filename,
			"application": data.Application,
			"platform": data.Platform,
		}).Info("Dump configuration")
	}
}

// confPath give the path of the configuration file given an application context
// It returns the absolute path to the file
func (hap *Haproxy) confPath() string {
	baseDir := hap.properties.HapHome + "/" + hap.Application + "/Config"
	os.MkdirAll(baseDir, 0755)
	return baseDir + "/hap" + hap.Application + hap.Platform + ".conf"
}

// confPath give the path of the archived configuration file given an application context
func (hap *Haproxy) confArchivePath() string {
	baseDir := hap.properties.HapHome + "/" + hap.Application + "/version-1"
	// It returns the absolute path to the file
	os.MkdirAll(baseDir, 0755)
	return baseDir + "/hap" + hap.Application + hap.Platform + ".conf"
}

// NewErrorPath gives a unique path the error file given the hap context
// It returns the full path to the file
func (hap *Haproxy) NewErrorPath() string {
	baseDir := hap.properties.HapHome + "/" + hap.Application + "/errors"
	os.MkdirAll(baseDir, 0755)
	prefix := time.Now().Format("20060102150405")
	return baseDir + "/" + prefix + "_" + hap.Application + hap.Platform + ".log"
}

func (hap *Haproxy) NewDebugPath() string {
	baseDir := hap.properties.HapHome + "/" + hap.Application + "/dump"
	os.MkdirAll(baseDir, 0755)
	prefix := time.Now().Format("20060102150405")
	return baseDir + "/" + prefix + "_" + hap.Application + hap.Platform + ".log"
}

// reload calls external shell script to reload haproxy
// It returns error if the reload fails
func (hap *Haproxy) reload(correlationId string) error {

	reloadScript := hap.getReloadScript()

	cmd := exec.Command("sh", reloadScript, "reload", "-y")
	env := os.Environ()
	env = append(env, "HAP_INTERACTIVE=0")
	cmd.Env = env
	output, err := cmd.Output()

	if err != nil {
		log.WithError(err).Error("Error reloading")
	}
	log.WithFields(log.Fields{
		"correlationId" : correlationId,
		"application": hap.Application,
		"platform": hap.Platform,
		"reloadScript": reloadScript,
	}).WithField("cmd", cmd).Debug("Reload succeeded")
	return err
}

// rollbac reverts configuration files and call for reload
func (hap *Haproxy) rollback(correlationId string) error {
	lastConf := hap.confArchivePath()
	if _, err := os.Stat(lastConf); os.IsNotExist(err) {
		return errors.New("No configuration file to rollback")
	}
	os.Rename(lastConf, hap.confPath())
	hap.reload(correlationId)
	return nil
}

// createSkeleton creates the directory tree for a new haproxy context
func (hap *Haproxy) createSkeleton(correlationId string) error {
	baseDir := hap.properties.HapHome + "/" + hap.Application

	createDirectory(correlationId, baseDir + "/Config")
	createDirectory(correlationId, baseDir + "/logs/" + hap.Application + hap.Platform)
	createDirectory(correlationId, baseDir + "/scripts")
	createDirectory(correlationId, baseDir + "/version-1")

	updateSymlink(correlationId, hap.getHapctlFilename(), hap.getReloadScript())
	updateSymlink(correlationId, hap.getHapBinary(), baseDir + "/Config/haproxy")

	log.WithFields(log.Fields{
		"correlationId" : correlationId,
		"dir": baseDir,
	}).Info("Skeleton created")

	return nil
}

// confPath give the path of the configuration file given an application context
// It returns the absolute path to the file
func (hap *Haproxy) syslogFragmentPath() string {
	baseDir := hap.properties.HapHome + "/SYSLOG/Config/syslog.conf.d"
	os.MkdirAll(baseDir, 0755)
	return baseDir + "/syslog" + hap.Application + hap.Platform + ".conf"
}

// updateSymlink create or update a symlink
func updateSymlink(correlationId, oldname string, newname string) {
	if _, err := os.Stat(newname); err == nil {
		os.Remove(newname)
	}
	err := os.Symlink(oldname, newname)
	if err != nil {
		log.WithError(err).WithFields(log.Fields{
			"correlationId" : correlationId,
			"path": newname,
		}).Error("Failed to create symlink")
	}
}

// createDirectory recursively creates directory if it doesn't exists
func createDirectory(correlationId string, dir string) {
	if _, err := os.Stat(dir); os.IsNotExist(err) {
		err := os.MkdirAll(dir, 0755)
		if err != nil {
			log.WithError(err).WithFields(log.Fields{
				"correlationId" : correlationId,
				"dir": dir,
			}).Error("Failed to create")
		} else {
			log.WithFields(log.Fields{
				"correlationId" : correlationId,
				"dir": dir,
			}).Info("Directory created")
		}
	}
}

// getHapctlFilename return the path to the vsc hapctl shell script
// This script is provided
func (hap *Haproxy) getHapctlFilename() string {
	return "/HOME/uxwadm/scripts/hapctl_unif"
}

// getReloadScript calculates reload script path given the hap context
// It returns the full script path
func (hap *Haproxy) getReloadScript() string {
	return fmt.Sprintf("%s/%s/scripts/hapctl%s%s", hap.properties.HapHome, hap.Application, hap.Application, hap.Platform)
}

// getHapBinary calculates the haproxy binary to use given the expected version
// It returns the full path to the haproxy binary
func (hap *Haproxy) getHapBinary() string {
	return fmt.Sprintf("/export/product/haproxy/product/%s/bin/haproxy", hap.Version)
}
<commit_msg>remove HAPROXY_INTERACTIVE and bug fix on output<commit_after>package haaasd

import (
	"bytes"
	"errors"
	"fmt"
	log "github.com/Sirupsen/logrus"
	"io/ioutil"
	"os"
	"os/exec"
	"time"
)

func NewHaproxy(properties *Config, application string, platform string, version string) *Haproxy {
	if version == "" {
		version = "1.4.22"
	}
	return &Haproxy{
		Application: application,
		Platform:    platform,
		properties:  properties,
		Version:     version,
	}
}

type Haproxy struct {
	Application string
	Platform    string
	Version     string
	properties  *Config
	State       int
}

const (
	SUCCESS int = iota
	UNCHANGED int = iota
	ERR_SYSLOG int = iota
	ERR_CONF int = iota
	ERR_RELOAD int = iota
)

// ApplyConfiguration write the new configuration and reload
// A rollback is called on failure
func (hap *Haproxy) ApplyConfiguration(data *EventMessage) (int, error) {
	hap.createSkeleton(data.Correlationid)

	newConf := data.Conf
	path := hap.confPath()

	// Check conf diff
	oldConf, err := ioutil.ReadFile(path)

	if err != nil {
		log.WithFields(log.Fields{
			"correlationId" : data.Correlationid,
			"path": path,
		}).Error("Cannot read old configuration")
		return ERR_CONF, err
	}
	if log.GetLevel() == log.DebugLevel {
		hap.dumpConfiguration(hap.NewDebugPath(), newConf, data)
	}
	if bytes.Equal(oldConf, newConf) {
		log.WithFields(log.Fields{
			"correlationId": data.Correlationid,
			"application": data.Application,
			"plateform":   data.Platform,
		}).Info("Ignore unchanged configuration")
		return UNCHANGED, nil
	}

	// Archive previous configuration
	archivePath := hap.confArchivePath()
	os.Rename(path, archivePath)
	log.WithFields(
		log.Fields{
			"correlationId": data.Correlationid,
			"application": data.Application,
			"plateform":   data.Platform,
			"archivePath": archivePath,
		}).Info("Old configuration saved")
	err = ioutil.WriteFile(path, newConf, 0644)
	if err != nil {
		return ERR_CONF, err
	}
	log.WithField("path", path).Info("New configuration written")

	// Reload haproxy
	err = hap.reload(data.Correlationid)
	if err != nil {
		log.WithFields(log.Fields{
			"application": data.Application,
			"plateform":   data.Platform,
		}).WithError(err).Error("Reload failed")
		hap.dumpConfiguration(hap.NewErrorPath(), newConf, data)
		err = hap.rollback(data.Correlationid)
		return ERR_RELOAD, err
	}
	// Write syslog fragment
	fragmentPath := hap.syslogFragmentPath()
	err = ioutil.WriteFile(fragmentPath, data.SyslogFragment, 0644)
	if err != nil {
		log.WithFields(log.Fields{
			"application": data.Application,
			"plateform":   data.Platform,
		}).WithError(err).Error("Failed to write syslog fragment")
		// TODO Should we rollback on syslog error ?
		return ERR_SYSLOG, err
	}

	return SUCCESS, nil
}

// dumpConfiguration dumps the new configuration file with context for debugging purpose
func (hap *Haproxy) dumpConfiguration(filename string, newConf []byte, data *EventMessage) {

	f, err2 := os.Create(filename)
	defer f.Close()
	if err2 == nil {
		f.WriteString("================================================================\n")
		f.WriteString(fmt.Sprintf("application: %s\n", data.Application))
		f.WriteString(fmt.Sprintf("platform: %s\n", data.Platform))
		f.WriteString(fmt.Sprintf("correlationid: %s\n", data.Correlationid))
		f.WriteString("================================================================\n")
		f.Write(newConf)
		f.Sync()

		log.WithFields(log.Fields{
			"correlationId": data.Correlationid,
			"filename": filename,
			"application": data.Application,
			"platform": data.Platform,
		}).Info("Dump configuration")
	}
}

// confPath give the path of the configuration file given an application context
// It returns the absolute path to the file
func (hap *Haproxy) confPath() string {
	baseDir := hap.properties.HapHome + "/" + hap.Application + "/Config"
	os.MkdirAll(baseDir, 0755)
	return baseDir + "/hap" + hap.Application + hap.Platform + ".conf"
}

// confPath give the path of the archived configuration file given an application context
func (hap *Haproxy) confArchivePath() string {
	baseDir := hap.properties.HapHome + "/" + hap.Application + "/version-1"
	// It returns the absolute path to the file
	os.MkdirAll(baseDir, 0755)
	return baseDir + "/hap" + hap.Application + hap.Platform + ".conf"
}

// NewErrorPath gives a unique path the error file given the hap context
// It returns the full path to the file
func (hap *Haproxy) NewErrorPath() string {
	baseDir := hap.properties.HapHome + "/" + hap.Application + "/errors"
	os.MkdirAll(baseDir, 0755)
	prefix := time.Now().Format("20060102150405")
	return baseDir + "/" + prefix + "_" + hap.Application + hap.Platform + ".log"
}

func (hap *Haproxy) NewDebugPath() string {
	baseDir := hap.properties.HapHome + "/" + hap.Application + "/dump"
	os.MkdirAll(baseDir, 0755)
	prefix := time.Now().Format("20060102150405")
	return baseDir + "/" + prefix + "_" + hap.Application + hap.Platform + ".log"
}

// reload calls external shell script to reload haproxy
// It returns error if the reload fails
func (hap *Haproxy) reload(correlationId string) error {

	reloadScript := hap.getReloadScript()
	cmd, err := exec.Command("sh", reloadScript, "reload", "-y").Output()
	if err != nil {
		log.WithError(err).Error("Error reloading")
	}
	log.WithFields(log.Fields{
		"correlationId" : correlationId,
		"application": hap.Application,
		"platform": hap.Platform,
		"reloadScript": reloadScript,
	}).WithField("cmd", cmd).Debug("Reload succeeded")
	return err
}

// rollbac reverts configuration files and call for reload
func (hap *Haproxy) rollback(correlationId string) error {
	lastConf := hap.confArchivePath()
	if _, err := os.Stat(lastConf); os.IsNotExist(err) {
		return errors.New("No configuration file to rollback")
	}
	os.Rename(lastConf, hap.confPath())
	hap.reload(correlationId)
	return nil
}

// createSkeleton creates the directory tree for a new haproxy context
func (hap *Haproxy) createSkeleton(correlationId string) error {
	baseDir := hap.properties.HapHome + "/" + hap.Application

	createDirectory(correlationId, baseDir + "/Config")
	createDirectory(correlationId, baseDir + "/logs/" + hap.Application + hap.Platform)
	createDirectory(correlationId, baseDir + "/scripts")
	createDirectory(correlationId, baseDir + "/version-1")

	updateSymlink(correlationId, hap.getHapctlFilename(), hap.getReloadScript())
	updateSymlink(correlationId, hap.getHapBinary(), baseDir + "/Config/haproxy")

	log.WithFields(log.Fields{
		"correlationId" : correlationId,
		"dir": baseDir,
	}).Info("Skeleton created")

	return nil
}

// confPath give the path of the configuration file given an application context
// It returns the absolute path to the file
func (hap *Haproxy) syslogFragmentPath() string {
	baseDir := hap.properties.HapHome + "/SYSLOG/Config/syslog.conf.d"
	os.MkdirAll(baseDir, 0755)
	return baseDir + "/syslog" + hap.Application + hap.Platform + ".conf"
}

// updateSymlink create or update a symlink
func updateSymlink(correlationId, oldname string, newname string) {
	if _, err := os.Stat(newname); err == nil {
		os.Remove(newname)
	}
	err := os.Symlink(oldname, newname)
	if err != nil {
		log.WithError(err).WithFields(log.Fields{
			"correlationId" : correlationId,
			"path": newname,
		}).Error("Failed to create symlink")
	}
}

// createDirectory recursively creates directory if it doesn't exists
func createDirectory(correlationId string, dir string) {
	if _, err := os.Stat(dir); os.IsNotExist(err) {
		err := os.MkdirAll(dir, 0755)
		if err != nil {
			log.WithError(err).WithFields(log.Fields{
				"correlationId" : correlationId,
				"dir": dir,
			}).Error("Failed to create")
		} else {
			log.WithFields(log.Fields{
				"correlationId" : correlationId,
				"dir": dir,
			}).Info("Directory created")
		}
	}
}

// getHapctlFilename return the path to the vsc hapctl shell script
// This script is provided
func (hap *Haproxy) getHapctlFilename() string {
	return "/HOME/uxwadm/scripts/hapctl_unif"
}

// getReloadScript calculates reload script path given the hap context
// It returns the full script path
func (hap *Haproxy) getReloadScript() string {
	return fmt.Sprintf("%s/%s/scripts/hapctl%s%s", hap.properties.HapHome, hap.Application, hap.Application, hap.Platform)
}

// getHapBinary calculates the haproxy binary to use given the expected version
// It returns the full path to the haproxy binary
func (hap *Haproxy) getHapBinary() string {
	return fmt.Sprintf("/export/product/haproxy/product/%s/bin/haproxy", hap.Version)
}
<|endoftext|>