<commit_before>package main

import (
	"fmt"
)

// Push command line tool
func Push() int {

	return 0
}

func printPushHelp() {
	fmt.Println(`Usage: git-lob push [options] [<remote> [<ref>...]]

  Uploads binaries to a remote, sending only binaries required to ensure
  that remote has the binary resources referenced at a set of commits.

  Behaves much like 'git push' except there are no destination refs, only
  supporting binary files.

Parameters:
  <remote>: The destination to upload to. This should correspond to the
            name of a remote (no direct URLs permitted) which is configured
            in .git/config. See REMOTES below for more details, additional
            config parameters are required in the remote.

            If no remote is specified, branch.*.remote configuration for the
            current branch is consulted to determine where to push. If the
            configuration is missing, it defaults to origin.
     <ref>: Which local reference(s) up to which we should make sure binaries
            are uploaded for. You can specify zero, one, or many local refs.
            There is no destination ref as in git push.

            If no ref is specified, and --all is not used, remote.*.push
            is consulted to see what to push, with push.default as a fallback.

Options:
  --all            Push all branches; cannot be used with other refs.
  --quiet, -q      Print less output
  --verbose, -v    Print more output
  --dry-run        Don't actually delete anything, just report

REMOTES

Binaries in git-lob are not stored in the regular git repo, but a corresponding
binary store must always exist to provide the actual binary content. A remote
in git usually only gives you the real git repo, so git-lob needs to expand
the configuration parameters to git remotes to specify the location of the
corresponding remote binary store.

The <remote> parameter refers to a named remote in .git/config (plain URLs
cannot be supported). The remote entry itself is the same as any normal git
remote, except that it requires additional git-lob specific parameters. The
parameters depend on the type of binary storage ('provider') being used; see
'git-lob providers' for a list of available providers and
'git-lob provider <provider>' for specific details of one provider.

As an example, let's take the 'filesystem' provider, which simply uses the OS's
file system as a remote transport (obviously very simplistic):

[remote "origin"]
    # these 2 lines are standard git
    url = ssh://git@bitbucket.org/something/somthing.git
    fetch = +refs/heads/*:refs/remotes/origin/*
    # these next 2 lines are required to configure the remote binary store
    git-lob-provider = filesystem
    git-lob-url = /Volumes/shared/something/something/binary/store

Other providers may require other parameters. It's important to note that you
can share a binary store among multiple remote repos if you wish, much like
the local git-lob.sharedstore option, since binaries are stored by SHA.
Identical file content in multiple repos can be stored only once this way.
Of course, access control may be an issue to consider here though.
`)
}
<commit_msg>Added details of the history checking to the push man entry<commit_after>package main

import (
	"fmt"
)

// Push command line tool
func Push() int {

	return 0
}

func printPushHelp() {
	fmt.Println(`Usage: git-lob push [options] [<remote> [<ref>...]]

  Uploads binaries to a remote, sending only binaries required to ensure
  that remote has the binary resources referenced at a set of commits.

  Behaves much like 'git push' except there are no destination refs, only
  supporting binary files.

Parameters:
  <remote>: The destination to upload to. This should correspond to the
            name of a remote (no direct URLs permitted) which is configured
            in .git/config. See REMOTES below for more details, additional
            config parameters are required in the remote.

            If no remote is specified, branch.*.remote configuration for the
            current branch is consulted to determine where to push. If the
            configuration is missing, it defaults to origin.
     <ref>: Which local reference(s) up to which we should make sure binaries
            are uploaded for. You can specify zero, one, or many local refs.
            There is no destination ref as in git push.

            If no ref is specified, and --all is not used, remote.*.push
            is consulted to see what to push, with push.default as a fallback.

            COMMIT RANGES

            You can also specify a range of refs in the form <ref1>..<ref2> to
            force git-lob push to check a specific range of commits for
            binaries, instead of using its own records of which commits it
            thinks are already up to date on this remote.
            See HISTORY CHECKING below.

Options:
  --all         Push all branches; cannot be used with other refs.
  --recheck     Re-check entire commit history to each ref instead of only
                back to last commit we believe is already pushed.
                See HISTORY CHECKING below for more details.
  --force       Always upload files even if the provider believes the file is
                already present on the remote. You shouldn't need this.
  --quiet, -q   Print less output
  --verbose, -v Print more output
  --dry-run     Don't actually delete anything, just report

REMOTES

Binaries in git-lob are not stored in the regular git repo, but a corresponding
binary store must always exist to provide the actual binary content. A remote
in git usually only gives you the real git repo, so git-lob needs to expand
the configuration parameters to git remotes to specify the location of the
corresponding remote binary store.

The <remote> parameter refers to a named remote in .git/config (plain URLs
cannot be supported). The remote entry itself is the same as any normal git
remote, except that it requires additional git-lob specific parameters. The
parameters depend on the type of binary storage ('provider') being used; see
'git-lob providers' for a list of available providers and
'git-lob provider <provider>' for specific details of one provider.

As an example, let's take the 'filesystem' provider, which simply uses the OS's
file system as a remote transport (obviously very simplistic):

[remote "origin"]
    # these 2 lines are standard git
    url = ssh://git@bitbucket.org/something/somthing.git
    fetch = +refs/heads/*:refs/remotes/origin/*
    # these next 2 lines are required to configure the remote binary store
    git-lob-provider = filesystem
    git-lob-url = /Volumes/shared/something/something/binary/store

Other providers may require other parameters. It's important to note that you
can share a binary store among multiple remote repos if you wish, much like
the local git-lob.sharedstore option, since binaries are stored by SHA.
Identical file content in multiple repos can be stored only once this way.
Of course, access control may be an issue to consider here though.

HISTORY CHECKING

When pushing binaries for a given ref, git-lob performs a search for commits
which reference git-lob binaries from that ref backwards, before checking
which of those binaries it needs to upload. This is so that we only upload
binaries that are actually referenced by the ref you're choosing to push,
and don't waste time on binaries in unpublished feature branches etc.

Because searching the whole of the git history can be slow on large
repositories, git-lob speeds this search up by keeping a record of which
commits it believes the remote already has all binaries for.

These records are updated whenever you git-lob push/pull. We do not use git's
own remote branch refs to track this, because pushing commits can be done
completely separately from binaries so we can't rely on that information.
So pushing and pulling branches in git has no effect on this state, only
git-lob push/pull.

If for some reason these records are wrong, and you need to push binaries
for a bigger range of commits, you can do this 2 ways:

1. Use the --recheck option. This is the 'nuclear option'; it will scan the
   entire history of the repo again to make 100% sure everything is correct.
   Can take a while on large repos.

2. Use a commit range for <ref>, i.e. <ref1>..<ref2>. git-lob will check that
   entire range of commits for binary references which will then be checked
   with the remote.

There are not many circumstances where you need to manually override the commit
range that is checked for binaries. Even if you edit commits, rebase etc,
git-lob should not miss any binaries, because the commit SHAs would change and
it would know to check any referenced binaries again. The main reason why
you would need to override the history checking is if the remote changed, for
example if someone manually deleted the remote binary store, or you moved to
a new URL without copying the data and needed to re-populate it from your local
repo.

`)
}
<|endoftext|>


<commit_before>package main

import (
	"fmt"
	"net/http"
	"time"

	"github.com/lxc/lxd"
)

func cmdShutdown() error {
	var timeout int

	if *argTimeout == -1 {
		timeout = 60
	} else {
		timeout = *argTimeout
	}

	c, err := lxd.NewClient(&lxd.DefaultConfig, "local")
	if err != nil {
		return err
	}

	req, err := http.NewRequest("PUT", c.BaseURL+"/internal/shutdown", nil)
	if err != nil {
		return err
	}

	_, err = c.Http.Do(req)
	if err != nil {
		return err
	}

	monitor := make(chan error, 1)
	go func() {
		monitor <- c.Monitor(nil, func(m interface{}) {}, nil)
	}()

	select {
	case <-monitor:
		break
	case <-time.After(time.Second * time.Duration(timeout)):
		return fmt.Errorf("LXD still running after %ds timeout.", timeout)
	}

	return nil
}
<commit_msg>Port main_shutdown to new client<commit_after>package main

import (
	"fmt"
	"time"

	"github.com/lxc/lxd/client"
)

func cmdShutdown() error {
	var timeout int

	if *argTimeout == -1 {
		timeout = 60
	} else {
		timeout = *argTimeout
	}

	c, err := lxd.ConnectLXDUnix("", nil)
	if err != nil {
		return err
	}

	_, _, err = c.RawQuery("PUT", "/internal/shutdown", nil, "")
	if err != nil {
		return err
	}

	chMonitor := make(chan bool, 1)
	go func() {
		monitor, err := c.GetEvents()
		if err != nil {
			close(chMonitor)
			return
		}

		monitor.Wait()
		close(chMonitor)
	}()

	select {
	case <-chMonitor:
		break
	case <-time.After(time.Second * time.Duration(timeout)):
		return fmt.Errorf("LXD still running after %ds timeout.", timeout)
	}

	return nil
}
<|endoftext|>


<commit_before>// Copyright © 2016 Patrick Young <patrick.mckendree.young@gmail.com>
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.

package cmd

import (
	"encoding/json"
	"fmt"

	"github.com/spf13/cobra"
)

func orderStatus(cmd *cobra.Command, args []string) (err error) {

	// Aquire an Api.
	api, err := apiFromConfig()
	if err != nil {
		return err
	}

	var orders Orders
	for _, id := range args {
		order, err := api.OrderStatus(id)
		if err != nil {
			return err
		}
		orders.Append(order)
	}

	result, err := json.Marshal(orders)
	if err != nil {
		return err
	}
	fmt.Printf("%s\n", result)
	return cacheToken(api)
}

// orderCmd represents the order command
var orderStatusCmd = &cobra.Command{
	Use:   "status [ORDERIDs]",
	Short: "Check the status of GBDX orders",
	Long: `Check the status of GBDX orders.

Pass the order ids in space delimited arguments to check multiple
orders.`,
	RunE: orderStatus,
}

func init() {
	orderCmd.AddCommand(orderStatusCmd)
}
<commit_msg>Order status now concurrent.<commit_after>// Copyright © 2016 Patrick Young <patrick.mckendree.young@gmail.com>
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.

package cmd

import (
	"bufio"
	"encoding/json"
	"fmt"
	"os"

	"github.com/spf13/cobra"
	"github.com/youngpm/gbdx"
)

func orderStatus(cmd *cobra.Command, args []string) (err error) {

	// Read order ids from stdin (line seperated)  if given no arguments.
	if len(args) == 0 {
		scanner := bufio.NewScanner(os.Stdin)
		for scanner.Scan() {
			args = append(args, scanner.Text())
		}
	}

	// We do things concurrently, so make a channel for returning
	// responses on and one to use as a counting semaphore.
	type orderResponse struct {
		order *gbdx.Order
		err   error
	}
	ch := make(chan orderResponse, len(args))
	sema := make(chan struct{}, 10) // Max concurrent requests.

	// Aquire an Api.
	api, err := apiFromConfig()
	if err != nil {
		return err
	}

	// Concurrently fetch the order status.
	for _, id := range args {
		go func(id string) {
			sema <- struct{}{}
			var o orderResponse
			o.order, o.err = api.OrderStatus(id)
			<-sema
			ch <- o
		}(id)
	}

	// Aggregate the results.
	var orders Orders
	for range args {
		o := <-ch
		if o.err != nil {
			return o.err
		}
		orders.Append(o.order)
	}

	// Marshall the aggregated result.
	result, err := json.Marshal(orders)
	if err != nil {
		return err
	}
	fmt.Printf("%s\n", result)
	return cacheToken(api)
}

// orderCmd represents the order command
var orderStatusCmd = &cobra.Command{
	Use:   "status [ORDERIDs]",
	Short: "Check the status of GBDX orders",
	Long: `Check the status of GBDX orders.

Pass the order ids in space delimited arguments to check multiple
orders, or if given no arguments, passed in delimited by newlines via
stdin.

The requests are done concurrently for speed.`,
	RunE: orderStatus,
}

func init() {
	orderCmd.AddCommand(orderStatusCmd)
}
<|endoftext|>


<commit_before>// Copyright 2013 Matthew Baird
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//     http://www.apache.org/licenses/LICENSE-2.0
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package elastigo

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"io/ioutil"
	"log"
	"net/http/httputil"
	"time"
)

func (c *Conn) DoCommand(method string, url string, args map[string]interface{}, data interface{}) ([]byte, error) {
	var response map[string]interface{}
	var body []byte
	var httpStatusCode int

	query, err := Escape(args)
	if err != nil {
		return nil, err
	}
	req, err := c.NewRequest(method, url, query)
	if err != nil {
		return body, err
	}

	if data != nil {
		switch v := data.(type) {
		case string:
			req.SetBodyString(v)
		case io.Reader:
			req.SetBody(v)
		case []byte:
			req.SetBodyBytes(v)
		default:
			err = req.SetBodyJson(v)
			if err != nil {
				return body, err
			}
		}
	}

	// uncomment this to print out the request that hits the wire
	//reqbuf, err := httputil.DumpRequest(req.Request, true)
	//log.Println(fmt.Sprintf("\n========= req:\nURL: %s\n%s", req.URL, bytes.NewBuffer(reqbuf).String()))

	// Copy request body for tracer
	if c.RequestTracer != nil {
		requestBody, err := ioutil.ReadAll(req.Body)
		if err != nil {
			return body, err
		}

		req.SetBody(bytes.NewReader(requestBody))
		c.RequestTracer(req.Method, req.URL.String(), string(requestBody))
	}

	httpStatusCode, body, err = req.Do(&response)
	if err != nil {
		return body, err
	}
	if httpStatusCode > 304 {

		jsonErr := json.Unmarshal(body, &response)
		if jsonErr == nil {
			if res_err, ok := response["error"]; ok {
				status, _ := response["status"]
				return body, ESError{time.Now(), fmt.Sprintf("Error [%s] Status [%v]", res_err, status), httpStatusCode}
			}
		}
		return body, jsonErr
	}
	return body, nil
}

// ESError is an error implementation that includes a time, message, and code.
type ESError struct {
	When time.Time
	What string
	Code int
}

func (e ESError) Error() string {
	return fmt.Sprintf("%v: %v [%v]", e.When, e.What, e.Code)
}

// Exists allows the caller to check for the existance of a document using HEAD
// This appears to be broken in the current version of elasticsearch 0.19.10, currently
// returning nothing
func (c *Conn) Exists(index string, _type string, id string, args map[string]interface{}) (BaseResponse, error) {
	var response map[string]interface{}
	var body []byte
	var url string
	var retval BaseResponse
	var httpStatusCode int

	query, err := Escape(args)
	if err != nil {
		return retval, err
	}

	if len(_type) > 0 {
		url = fmt.Sprintf("/%s/%s/%s", index, _type, id)
	} else {
		url = fmt.Sprintf("/%s/%s", index, id)
	}
	req, err := c.NewRequest("HEAD", url, query)
	if err != nil {
		// some sort of generic error handler
	}
	httpStatusCode, body, err = req.Do(&response)
	if httpStatusCode > 304 {
		if error, ok := response["error"]; ok {
			status, _ := response["status"]
			log.Printf("Error: %v (%v)\n", error, status)
		}
	} else {
		// marshall into json
		jsonErr := json.Unmarshal(body, &retval)
		if jsonErr != nil {
			log.Println(jsonErr)
		}
	}
	return retval, err
}
<commit_msg>Fixed broken import<commit_after>// Copyright 2013 Matthew Baird
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//     http://www.apache.org/licenses/LICENSE-2.0
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package elastigo

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"io/ioutil"
	"log"
	//"net/http/httputil"
	"time"
)

func (c *Conn) DoCommand(method string, url string, args map[string]interface{}, data interface{}) ([]byte, error) {
	var response map[string]interface{}
	var body []byte
	var httpStatusCode int

	query, err := Escape(args)
	if err != nil {
		return nil, err
	}
	req, err := c.NewRequest(method, url, query)
	if err != nil {
		return body, err
	}

	if data != nil {
		switch v := data.(type) {
		case string:
			req.SetBodyString(v)
		case io.Reader:
			req.SetBody(v)
		case []byte:
			req.SetBodyBytes(v)
		default:
			err = req.SetBodyJson(v)
			if err != nil {
				return body, err
			}
		}
	}

	// uncomment this to print out the request that hits the wire
	//   (requires net/http/httputil)
	//reqbuf, err := httputil.DumpRequest(req.Request, true)
	//log.Println(fmt.Sprintf("\n========= req:\nURL: %s\n%s", req.URL, bytes.NewBuffer(reqbuf).String()))

	// Copy request body for tracer
	if c.RequestTracer != nil {
		requestBody, err := ioutil.ReadAll(req.Body)
		if err != nil {
			return body, err
		}

		req.SetBody(bytes.NewReader(requestBody))
		c.RequestTracer(req.Method, req.URL.String(), string(requestBody))
	}

	httpStatusCode, body, err = req.Do(&response)
	if err != nil {
		return body, err
	}
	if httpStatusCode > 304 {

		jsonErr := json.Unmarshal(body, &response)
		if jsonErr == nil {
			if res_err, ok := response["error"]; ok {
				status, _ := response["status"]
				return body, ESError{time.Now(), fmt.Sprintf("Error [%s] Status [%v]", res_err, status), httpStatusCode}
			}
		}
		return body, jsonErr
	}
	return body, nil
}

// ESError is an error implementation that includes a time, message, and code.
type ESError struct {
	When time.Time
	What string
	Code int
}

func (e ESError) Error() string {
	return fmt.Sprintf("%v: %v [%v]", e.When, e.What, e.Code)
}

// Exists allows the caller to check for the existance of a document using HEAD
// This appears to be broken in the current version of elasticsearch 0.19.10, currently
// returning nothing
func (c *Conn) Exists(index string, _type string, id string, args map[string]interface{}) (BaseResponse, error) {
	var response map[string]interface{}
	var body []byte
	var url string
	var retval BaseResponse
	var httpStatusCode int

	query, err := Escape(args)
	if err != nil {
		return retval, err
	}

	if len(_type) > 0 {
		url = fmt.Sprintf("/%s/%s/%s", index, _type, id)
	} else {
		url = fmt.Sprintf("/%s/%s", index, id)
	}
	req, err := c.NewRequest("HEAD", url, query)
	if err != nil {
		// some sort of generic error handler
	}
	httpStatusCode, body, err = req.Do(&response)
	if httpStatusCode > 304 {
		if error, ok := response["error"]; ok {
			status, _ := response["status"]
			log.Printf("Error: %v (%v)\n", error, status)
		}
	} else {
		// marshall into json
		jsonErr := json.Unmarshal(body, &retval)
		if jsonErr != nil {
			log.Println(jsonErr)
		}
	}
	return retval, err
}
<|endoftext|>


<commit_before>package healthy

import (
	"bytes"
	"fmt"
	"text/template"

	"github.com/Nitro/sidecar/discovery"
	"github.com/Nitro/sidecar/service"
	log "github.com/sirupsen/logrus"
	"github.com/relistan/go-director"
)

const (
	DEFAULT_STATUS_ENDPOINT = "/"
)

func (m *Monitor) Services() []service.Service {
	var svcList []service.Service

	if m.DiscoveryFn == nil {
		log.Errorf("Error: DiscoveryFn not defined!")
		return []service.Service{}
	}

	for _, svc := range m.DiscoveryFn() {
		if svc.ID == "" {
			log.Errorf("Error: monitor found empty service ID")
			continue
		}

		m.MarkService(&svc)
		svcList = append(svcList, svc)
	}

	return svcList
}

func findFirstTCPPort(svc *service.Service) *service.Port {
	for _, port := range svc.Ports {
		if port.Type == "tcp" {
			return &port
		}
	}
	return nil
}

// Configure a default check for a service. The default is to return an HTTP
// check on the first TCP port on the endpoint set in DEFAULT_STATUS_ENDPOINT.
func (m *Monitor) defaultCheckForService(svc *service.Service) *Check {
	port := findFirstTCPPort(svc)
	if port == nil {
		return &Check{ID: svc.ID, Command: &AlwaysSuccessfulCmd{}}
	}

	// Use the const default unless we've been provided something else
	defaultCheckEndpoint := DEFAULT_STATUS_ENDPOINT
	if len(m.DefaultCheckEndpoint) != 0 {
		defaultCheckEndpoint = m.DefaultCheckEndpoint
	}

	url := fmt.Sprintf("http://%v:%v%v", m.DefaultCheckHost, port.Port, defaultCheckEndpoint)
	return &Check{
		ID:      svc.ID,
		Type:    "HttpGet",
		Args:    url,
		Status:  FAILED,
		Command: &HttpGetCmd{},
	}
}

func (m *Monitor) GetCommandNamed(name string) Checker {
	switch name {
	case "HttpGet":
		return &HttpGetCmd{}
	case "External":
		return &ExternalCmd{}
	case "AlwaysSuccessful":
		return &AlwaysSuccessfulCmd{}
	default:
		return &HttpGetCmd{}
	}
}

// Talks to a Discoverer and returns the configured check
func (m *Monitor) fetchCheckForService(svc *service.Service, disco discovery.Discoverer) *Check {

	check := &Check{}
	check.Type, check.Args = disco.HealthCheck(svc)
	if check.Type == "" {
		log.Warnf(
			"Adding check for service %s (id: %s) failed. Got check type: %s, check args: %s.",
			svc.Name, svc.ID, check.Type, check.Args,
		)
		return nil
	}

	// Setup some other parts of the check that don't come from discovery
	check.ID = svc.ID
	check.Command = m.GetCommandNamed(check.Type)
	check.Status = FAILED

	return check
}

// Use templating to substitute in some info about the service.  Important because
// we won't know the actual Port that the container will bind to, for example.
func (m *Monitor) templateCheckArgs(check *Check, svc *service.Service) string {
	funcMap := template.FuncMap{
		"tcp":       func(p int64) int64 { return svc.PortForServicePort(p, "tcp") },
		"udp":       func(p int64) int64 { return svc.PortForServicePort(p, "udp") },
		"host":      func() string { return m.DefaultCheckHost },
		"container": func() string { return svc.Hostname },
	}

	t, err := template.New("check").Funcs(funcMap).Parse(check.Args)
	if err != nil {
		log.Errorf("Unable to parse check Args: '%s'", check.Args)
		return check.Args
	}

	var output bytes.Buffer
	t.Execute(&output, svc)

	return output.String()
}

// CheckForService returns a Check that has been properly configured for this
// particular service.
func (m *Monitor) CheckForService(svc *service.Service, disco discovery.Discoverer) *Check {
	check := m.fetchCheckForService(svc, disco)
	if check == nil { // We got nothing
		log.Warnf("Using default check for service %s (id: %s).", svc.Name, svc.ID)
		check = m.defaultCheckForService(svc)
	}

	check.Args = m.templateCheckArgs(check, svc)

	return check
}

// Watch loops over a list of services and adds checks for services we don't already
// know about. It then removes any checks for services which have gone away. All
// services are expected to be local to this node.
func (m *Monitor) Watch(disco discovery.Discoverer, looper director.Looper) {
	m.DiscoveryFn = disco.Services // Store this so we can use it from Services()

	looper.Loop(func() error {
		services := disco.Services()

		// Add checks when new services are found
		for _, svc := range services {
			if m.Checks[svc.ID] == nil {
				check := m.CheckForService(&svc, disco)
				if check.Command == nil {
					log.Errorf(
						"Attempted to add %s (id: %s) but no check configured!",
						svc.Name, svc.ID,
					)
				} else {
					m.AddCheck(check)
				}
			}
		}

		m.Lock()
		defer m.Unlock()
	OUTER:
		// We remove checks when encountering a missing service. This
		// prevents us from storing up checks forever. This is the only
		// way we'll find out about a service going away.
		for _, check := range m.Checks {
			for _, svc := range services {
				// Continue if we have a matching service/check pair
				if svc.ID == check.ID {
					continue OUTER
				}
			}

			// Remove checks for services that are no longer running
			delete(m.Checks, check.ID)
		}

		return nil
	})
}
<commit_msg>Fix logging in fetchCheckForService<commit_after>package healthy

import (
	"bytes"
	"fmt"
	"text/template"

	"github.com/Nitro/sidecar/discovery"
	"github.com/Nitro/sidecar/service"
	"github.com/relistan/go-director"
	log "github.com/sirupsen/logrus"
)

const (
	DEFAULT_STATUS_ENDPOINT = "/"
)

func (m *Monitor) Services() []service.Service {
	var svcList []service.Service

	if m.DiscoveryFn == nil {
		log.Errorf("Error: DiscoveryFn not defined!")
		return []service.Service{}
	}

	for _, svc := range m.DiscoveryFn() {
		if svc.ID == "" {
			log.Errorf("Error: monitor found empty service ID")
			continue
		}

		m.MarkService(&svc)
		svcList = append(svcList, svc)
	}

	return svcList
}

func findFirstTCPPort(svc *service.Service) *service.Port {
	for _, port := range svc.Ports {
		if port.Type == "tcp" {
			return &port
		}
	}
	return nil
}

// Configure a default check for a service. The default is to return an HTTP
// check on the first TCP port on the endpoint set in DEFAULT_STATUS_ENDPOINT.
func (m *Monitor) defaultCheckForService(svc *service.Service) *Check {
	port := findFirstTCPPort(svc)
	if port == nil {
		return &Check{ID: svc.ID, Command: &AlwaysSuccessfulCmd{}}
	}

	// Use the const default unless we've been provided something else
	defaultCheckEndpoint := DEFAULT_STATUS_ENDPOINT
	if len(m.DefaultCheckEndpoint) != 0 {
		defaultCheckEndpoint = m.DefaultCheckEndpoint
	}

	url := fmt.Sprintf("http://%v:%v%v", m.DefaultCheckHost, port.Port, defaultCheckEndpoint)
	return &Check{
		ID:      svc.ID,
		Type:    "HttpGet",
		Args:    url,
		Status:  FAILED,
		Command: &HttpGetCmd{},
	}
}

func (m *Monitor) GetCommandNamed(name string) Checker {
	switch name {
	case "HttpGet":
		return &HttpGetCmd{}
	case "External":
		return &ExternalCmd{}
	case "AlwaysSuccessful":
		return &AlwaysSuccessfulCmd{}
	default:
		return &HttpGetCmd{}
	}
}

// Talks to a Discoverer and returns the configured check
func (m *Monitor) fetchCheckForService(svc *service.Service, disco discovery.Discoverer) *Check {

	check := &Check{}
	check.Type, check.Args = disco.HealthCheck(svc)
	if check.Type == "" {
		log.Warnf("Got empty check type for service %s (id: %s) with args: %s!", svc.Name, svc.ID, check.Args)
		return nil
	}

	// Setup some other parts of the check that don't come from discovery
	check.ID = svc.ID
	check.Command = m.GetCommandNamed(check.Type)
	check.Status = FAILED

	return check
}

// Use templating to substitute in some info about the service.  Important because
// we won't know the actual Port that the container will bind to, for example.
func (m *Monitor) templateCheckArgs(check *Check, svc *service.Service) string {
	funcMap := template.FuncMap{
		"tcp":       func(p int64) int64 { return svc.PortForServicePort(p, "tcp") },
		"udp":       func(p int64) int64 { return svc.PortForServicePort(p, "udp") },
		"host":      func() string { return m.DefaultCheckHost },
		"container": func() string { return svc.Hostname },
	}

	t, err := template.New("check").Funcs(funcMap).Parse(check.Args)
	if err != nil {
		log.Errorf("Unable to parse check Args: '%s'", check.Args)
		return check.Args
	}

	var output bytes.Buffer
	t.Execute(&output, svc)

	return output.String()
}

// CheckForService returns a Check that has been properly configured for this
// particular service.
func (m *Monitor) CheckForService(svc *service.Service, disco discovery.Discoverer) *Check {
	check := m.fetchCheckForService(svc, disco)
	if check == nil { // We got nothing
		log.Warnf("Using default check for service %s (id: %s).", svc.Name, svc.ID)
		check = m.defaultCheckForService(svc)
	}

	check.Args = m.templateCheckArgs(check, svc)

	return check
}

// Watch loops over a list of services and adds checks for services we don't already
// know about. It then removes any checks for services which have gone away. All
// services are expected to be local to this node.
func (m *Monitor) Watch(disco discovery.Discoverer, looper director.Looper) {
	m.DiscoveryFn = disco.Services // Store this so we can use it from Services()

	looper.Loop(func() error {
		services := disco.Services()

		// Add checks when new services are found
		for _, svc := range services {
			if m.Checks[svc.ID] == nil {
				check := m.CheckForService(&svc, disco)
				if check.Command == nil {
					log.Errorf(
						"Attempted to add %s (id: %s) but no check configured!",
						svc.Name, svc.ID,
					)
				} else {
					m.AddCheck(check)
				}
			}
		}

		m.Lock()
		defer m.Unlock()
	OUTER:
		// We remove checks when encountering a missing service. This
		// prevents us from storing up checks forever. This is the only
		// way we'll find out about a service going away.
		for _, check := range m.Checks {
			for _, svc := range services {
				// Continue if we have a matching service/check pair
				if svc.ID == check.ID {
					continue OUTER
				}
			}

			// Remove checks for services that are no longer running
			delete(m.Checks, check.ID)
		}

		return nil
	})
}
<|endoftext|>


<commit_before>package raft

import (
	"github.com/iketheadore/raft/comm"
	"github.com/iketheadore/raft/logic"
)

type Raft struct {
	localServ logic.Server
	listener  comm.Listener
	sender    comm.Sender
	logic     *logic.Logic
}

func New(addr string) *Raft {
	r := &Raft{localServ: logic.Server{Addr: addr, Role: logic.Follower}, listener: comm.NewListener(addr)}
	r.listener.Run()
	r.logic = logic.New(r.localServ)
	r.logic.Subscribe(r.listener)
	return r
}

func (r *Raft) Connect(addr string) error {
	// if contains(r.others, addr) {
	// 	return errors.New("duplicate addr:" + addr)
	// }
	return r.logic.Connect(logic.Server{Addr: addr, Role: logic.Follower})
	// r.others = append(r.others, logic.Server{Addr: addr, Role: logic.Follower})
	// glog.Info("other len:", len(r.others))
	// glog.Info(r.others)
	// return nil
}

func (r *Raft) Run() {
	r.logic.Run()
}

func (r *Raft) ReplicateCmd(cmd comm.Command) {
	r.logic.ReplicateCmd(cmd)
}
<commit_msg>remove some comments<commit_after>package raft

import (
	"github.com/iketheadore/raft/comm"
	"github.com/iketheadore/raft/logic"
)

type Raft struct {
	localServ logic.Server
	listener  comm.Listener
	sender    comm.Sender
	logic     *logic.Logic
}

func New(addr string) *Raft {
	r := &Raft{localServ: logic.Server{Addr: addr, Role: logic.Follower}, listener: comm.NewListener(addr)}
	r.listener.Run()
	r.logic = logic.New(r.localServ)
	r.logic.Subscribe(r.listener)
	return r
}

func (r *Raft) Connect(addr string) error {
	return r.logic.Connect(logic.Server{Addr: addr, Role: logic.Follower})
}

func (r *Raft) Run() {
	r.logic.Run()
}

func (r *Raft) ReplicateCmd(cmd comm.Command) {
	r.logic.ReplicateCmd(cmd)
}
<|endoftext|>


<commit_before>// Package GobDB implements a persistant key-value store of
// gob-compatible types. This is accomplished with a light
// wrapper around leveldb and Go's gob encoding library.
package GobDB

import (
	"github.com/syndtr/goleveldb/leveldb"
	"github.com/syndtr/goleveldb/leveldb/util"
	"strconv"
)

// DB is a LevelDB wrapper that stores key-value pairs of
// gob-compatible types.
type DB struct {
	internal *leveldb.DB
	location string
	encoder  FilteredEncoder
	decoder  Decoder
	prepared bool
}

// At returns an unopened database at with given datafile.
func At(path string) *DB {
	return &DB{location: path}
}

// Open sets up the internal leveldb if not done already.
func (db *DB) Open() error {
	if db.IsOpen() {
		return nil
	}

	ret, err := leveldb.OpenFile(db.location, nil)
	if err == nil {
		db.internal = ret
		db.prepare()
	}
	return err
}

// IsOpen checks whether or not the database is open.
func (db DB) IsOpen() bool {
	return db.internal != nil
}

// Close tears down the internal leveldb, writing all contents
// to file.
func (db *DB) Close() {
	if db.IsOpen() {
		db.internal.Close()
		db.internal = nil
	}
}

// Put encodes given key and value through gob, inserting resulting
// byte slices into the database's internal leveldb.
func (db *DB) Put(key, value interface{}) error {
	// Encode key via gob, registering types if necessary.
	o1, err := db.encode(key)
	if err != nil {
		return err
	}

	// Form prefixed key.
	pkey := []byte("key:")
	pkey = append(pkey, o1...)

	// Encode value via gob, registering types if necessary.
	val, err := db.encode(value)
	if err != nil {
		return err
	}

	// Insert gobbed values into leveldb.
	return db.internal.Put(pkey, val, nil)
}

// Get ncodes given key via gob, fetches the corresponding
// value from within leveldb, and decodes that value into
// parameter two.
func (db *DB) Get(key, value interface{}) error {
	// Encode key via gob, registering its type if necessary.
	obj, err := db.encode(key)
	if err != nil {
		return err
	}

	// Form prefixed key.
	pkey := []byte("key:")
	pkey = append(pkey, obj...)

	// Fetch gob-encoded value.
	val, err := db.internal.Get(pkey, nil)
	if err != nil {
		return err
	}

	// Decode value into second paramater (which should be a
	// pointer)
	return db.decoder.Decode(val, value)
}

// Has encodes given key via gob and checks if the resulting
// byte slice exists in the database's internal leveldb.
func (db DB) Has(key interface{}) bool {
	// Encode key via gob, registering its type if necessary.
	obj, err := db.encode(key)
	if err != nil {
		return false
	}

	// Note: this is niave - should check error type.
	_, err = db.internal.Get(obj, nil)
	return err == nil
}

// Delete encodes given key via gob, deleting the resulting
// byte slice from the database's internal leveldb.
func (db *DB) Delete(key interface{}) error {
	// Encode key via gob, registering its type if necessary.
	obj, err := db.encode(key)
	if err != nil {
		return err
	}

	// Delete!
	return db.internal.Delete(obj, nil)
}

// Entries counts key-value pairs in the database.
func (db *DB) Entries() int {
	i := 0
	iter := db.internal.NewIterator(util.BytesPrefix([]byte("key:")), nil)
	for iter.Next() {
		i++
	}
	iter.Release()
	return i
}

// Reset erases caches and closes leveldb. This way, the db
// is forced to reload gobbed values as though it had just
// been opened for the first time.
func (db *DB) Reset() {
	db.Close()
	db.prepared = false
}

// Encodes given key via gob, registers its type if necessary,
// and routes any errors outward.
func (db *DB) encode(key interface{}) ([]byte, error) {
	err := db.Open()
	if err != nil {
		return []byte{}, err
	}

	// Gob encode key.
	def, obj, err := db.encoder.Encode(key)
	if err != nil {
		return []byte{}, err
	}

	// Register key with decoder.
	db.decoder.Register(append(def, obj...))

	// Register key type.
	err = db.registerType(def, obj)
	if err != nil {
		return []byte{}, err
	}

	// err = db.decoder.Decode(append(def, obj...), nil)
	// if err != nil {
	// 	return []byte{}, err
	// }

	return obj, nil
}

// When the database is opened for the first time, scrolls through
// all entries to form the same encoder that was used before the
// previous db.Close() call.
//
// Note that this is an expensive operation, scaling linearly with
// dataset size. Accordingly, you should utilize the open and close
// methods instead of initializing new DB objects all the time. For
// each initialization, the decoder is literally thrown in the
// garbage.
func (db *DB) prepare() error {
	if db.prepared == true {
		return nil
	}

	iter := db.internal.NewIterator(util.BytesPrefix([]byte("prep#")), nil)
	for iter.Next() {
		value := iter.Value()
		db.encoder.Encode(value)
		db.decoder.Register(value)
	}
	iter.Release()
	err := iter.Error()
	if err == nil {
		db.prepared = true
	}
	return err
}

func (db *DB) setPrepCount(value int) error {
	key := []byte("prep-count")
	data := []byte(strconv.Itoa(value))
	return db.internal.Put(key, data, nil)
}

func (db *DB) incPrepCount(i int) error {
	size := db.prepCount()
	if size == -1 {
		return db.setPrepCount(1)
	}
	return db.setPrepCount(size + i)
}

func (db *DB) prepCount() int {
	err := db.Open()
	if err != nil {
		return -1
	}

	val, err := db.internal.Get([]byte("prep-count"), nil)
	if err != nil {
		return 0
	}
	n, _ := strconv.Atoi(string(val))
	return n
}

// Checks if definition is present in current db.
func (db *DB) isPresent(def []byte) bool {
	err := db.Open()
	if err != nil {
		return false
	}

	key := []byte("prep:")
	key = append(key, def...)
	_, err = db.internal.Get(key, nil)
	return err == nil
}

// If not done already, registers type and example object in both
// the encoder and decoder.
func (db *DB) registerType(def, obj []byte) error {
	// Ensure that database is open.
	err := db.Open()
	if err != nil {
		return err
	}

	// Stop if type is already registered in database.
	if db.isPresent(def) {
		return nil
	}

	// Map prep#<n> to type definition bytes.
	k1 := []byte("prep#" + strconv.Itoa(db.prepCount()))
	err = db.incPrepCount(1)
	if err != nil {
		return err
	}
	v1 := []byte{}
	v1 = append(v1, def...)
	v1 = append(v1, obj...)
	err = db.internal.Put(k1, v1, nil)
	if err != nil {
		return err
	}

	// Map prep:<def> to empty string (for checking duplicate keys).
	k2 := []byte("prep:")
	k2 = append(k2, def...)
	v2 := []byte("")
	err = db.internal.Put(k2, v2, nil)
	if err != nil {
		db.internal.Delete(k1, nil)
		db.incPrepCount(-1)
		return err
	}
	return nil
}
<commit_msg>Fixed delete.<commit_after>// Package GobDB implements a persistant key-value store of
// gob-compatible types. This is accomplished with a light
// wrapper around leveldb and Go's gob encoding library.
package GobDB

import (
	"github.com/syndtr/goleveldb/leveldb"
	"github.com/syndtr/goleveldb/leveldb/util"
	"strconv"
)

// DB is a LevelDB wrapper that stores key-value pairs of
// gob-compatible types.
type DB struct {
	internal *leveldb.DB
	location string
	encoder  FilteredEncoder
	decoder  Decoder
	prepared bool
}

// At returns an unopened database at with given datafile.
func At(path string) *DB {
	return &DB{location: path}
}

// Open sets up the internal leveldb if not done already.
func (db *DB) Open() error {
	if db.IsOpen() {
		return nil
	}

	ret, err := leveldb.OpenFile(db.location, nil)
	if err == nil {
		db.internal = ret
		db.prepare()
	}
	return err
}

// IsOpen checks whether or not the database is open.
func (db DB) IsOpen() bool {
	return db.internal != nil
}

// Close tears down the internal leveldb, writing all contents
// to file.
func (db *DB) Close() {
	if db.IsOpen() {
		db.internal.Close()
		db.internal = nil
	}
}

// Put encodes given key and value through gob, inserting resulting
// byte slices into the database's internal leveldb.
func (db *DB) Put(key, value interface{}) error {
	// Encode key via gob, registering types if necessary.
	o1, err := db.encode(key)
	if err != nil {
		return err
	}

	// Form prefixed key.
	pkey := []byte("key:")
	pkey = append(pkey, o1...)

	// Encode value via gob, registering types if necessary.
	val, err := db.encode(value)
	if err != nil {
		return err
	}

	// Insert gobbed values into leveldb.
	return db.internal.Put(pkey, val, nil)
}

// Get ncodes given key via gob, fetches the corresponding
// value from within leveldb, and decodes that value into
// parameter two.
func (db *DB) Get(key, value interface{}) error {
	// Encode key via gob, registering its type if necessary.
	obj, err := db.encode(key)
	if err != nil {
		return err
	}

	// Form prefixed key.
	pkey := []byte("key:")
	pkey = append(pkey, obj...)

	// Fetch gob-encoded value.
	val, err := db.internal.Get(pkey, nil)
	if err != nil {
		return err
	}

	// Decode value into second paramater (which should be a
	// pointer)
	return db.decoder.Decode(val, value)
}

// Has encodes given key via gob and checks if the resulting
// byte slice exists in the database's internal leveldb.
func (db DB) Has(key interface{}) bool {
	// Encode key via gob, registering its type if necessary.
	obj, err := db.encode(key)
	if err != nil {
		return false
	}

	// Note: this is niave - should check error type.
	_, err = db.internal.Get(obj, nil)
	return err == nil
}

// Delete encodes given key via gob, deleting the resulting
// byte slice from the database's internal leveldb.
func (db *DB) Delete(key interface{}) error {
	// Encode key via gob, registering its type if necessary.
	obj, err := db.encode(key)
	if err != nil {
		return err
	}

	// Form key bytes.
	kbytes := []byte("key:")
	kbytes = append(kbytes, obj...)

	// Delete!
	return db.internal.Delete(kbytes, nil)
}

// Entries counts key-value pairs in the database.
func (db *DB) Entries() int {
	i := 0
	iter := db.internal.NewIterator(util.BytesPrefix([]byte("key:")), nil)
	for iter.Next() {
		i++
	}
	iter.Release()
	return i
}

// Reset erases caches and closes leveldb. This way, the db
// is forced to reload gobbed values as though it had just
// been opened for the first time.
func (db *DB) Reset() {
	db.Close()
	db.prepared = false
}

// Encodes given key via gob, registers its type if necessary,
// and routes any errors outward.
func (db *DB) encode(key interface{}) ([]byte, error) {
	err := db.Open()
	if err != nil {
		return []byte{}, err
	}

	// Gob encode key.
	def, obj, err := db.encoder.Encode(key)
	if err != nil {
		return []byte{}, err
	}

	// Register key with decoder.
	db.decoder.Register(append(def, obj...))

	// Register key type.
	err = db.registerType(def, obj)
	if err != nil {
		return []byte{}, err
	}

	// err = db.decoder.Decode(append(def, obj...), nil)
	// if err != nil {
	// 	return []byte{}, err
	// }

	return obj, nil
}

// When the database is opened for the first time, scrolls through
// all entries to form the same encoder that was used before the
// previous db.Close() call.
//
// Note that this is an expensive operation, scaling linearly with
// dataset size. Accordingly, you should utilize the open and close
// methods instead of initializing new DB objects all the time. For
// each initialization, the decoder is literally thrown in the
// garbage.
func (db *DB) prepare() error {
	if db.prepared == true {
		return nil
	}

	iter := db.internal.NewIterator(util.BytesPrefix([]byte("prep#")), nil)
	for iter.Next() {
		value := iter.Value()
		db.encoder.Encode(value)
		db.decoder.Register(value)
	}
	iter.Release()
	err := iter.Error()
	if err == nil {
		db.prepared = true
	}
	return err
}

func (db *DB) setPrepCount(value int) error {
	key := []byte("prep-count")
	data := []byte(strconv.Itoa(value))
	return db.internal.Put(key, data, nil)
}

func (db *DB) incPrepCount(i int) error {
	size := db.prepCount()
	if size == -1 {
		return db.setPrepCount(1)
	}
	return db.setPrepCount(size + i)
}

func (db *DB) prepCount() int {
	err := db.Open()
	if err != nil {
		return -1
	}

	val, err := db.internal.Get([]byte("prep-count"), nil)
	if err != nil {
		return 0
	}
	n, _ := strconv.Atoi(string(val))
	return n
}

// Checks if definition is present in current db.
func (db *DB) isPresent(def []byte) bool {
	err := db.Open()
	if err != nil {
		return false
	}

	key := []byte("prep:")
	key = append(key, def...)
	_, err = db.internal.Get(key, nil)
	return err == nil
}

// If not done already, registers type and example object in both
// the encoder and decoder.
func (db *DB) registerType(def, obj []byte) error {
	// Ensure that database is open.
	err := db.Open()
	if err != nil {
		return err
	}

	// Stop if type is already registered in database.
	if db.isPresent(def) {
		return nil
	}

	// Map prep#<n> to type definition bytes.
	k1 := []byte("prep#" + strconv.Itoa(db.prepCount()))
	err = db.incPrepCount(1)
	if err != nil {
		return err
	}
	v1 := []byte{}
	v1 = append(v1, def...)
	v1 = append(v1, obj...)
	err = db.internal.Put(k1, v1, nil)
	if err != nil {
		return err
	}

	// Map prep:<def> to empty string (for checking duplicate keys).
	k2 := []byte("prep:")
	k2 = append(k2, def...)
	v2 := []byte("")
	err = db.internal.Put(k2, v2, nil)
	if err != nil {
		db.internal.Delete(k1, nil)
		db.incPrepCount(-1)
		return err
	}
	return nil
}
<|endoftext|>


<commit_before>package lmdb

import (
	"errors"
	mdb "github.com/szferi/gomdb"
)

// Thread Safety
// 1) NOTLS mode is used exclusively, which allows read txns to freely migrate across
//    threads and for a single thread to maintain multiple read txns. This enables mostly
//    care-free use of read txns.
// 2) Most objects can be safely called by a single caller from a single thread, and usually it
//    only makes sense to have a single caller, except in the case of Database.
// 3) Most Database methods are thread-safe, and may be called concurrently, except for
//    Database.Close().
// 4) A write txn may only be used from the thread it was created on.
// 5) A read-only txn can move across threads, but it cannot be used concurrently from multiple
//    threads.
// 6) Iterator is not thread-safe, but it does not make sense to use it on any thread except the
//    thread that currently owns its associated txn.
//
//----------------------------------------------------------------------------------------
//
// Best practice:
// 1) Use iterators only in the txn that they are created
// 2) DO NOT modify the memory slice from GetNoCopy
// 3) Make sure all read/write txns are finished before Database.Close().

const (
	// There is no penalty for making this huge.
	// If you are on a 32-bit system, use Open2 and specify a smaller map size.
	MAP_SIZE_DEFAULT uint64 = 64 * 1024 * 1024 * 1024 * 1024 // 64TB
)

type RWTxnCreator interface {
	TransactionalRW(func(*ReadWriteTxn) error) error
}

type dryRunDummyError struct { }

func (d dryRunDummyError) Error() string {
	return "Dummy error of dry running a db transaction."
}

// Dry run the db transaction, i.e. always rollback, even if the returned error is nil.
func DryRunRWTransaction(rwtxer RWTxnCreator, f func(*ReadWriteTxn) error) error {
	err := rwtxer.TransactionalRW(func(rwtx *ReadWriteTxn) error {
		err := f(rwtx)
		if err == nil {
			err = dryRunDummyError{}
		}
		return err
	})

	if _, ok := err.(dryRunDummyError); ok {
		err = nil
	}
	return err
}

//--------------------------------- Database ---------------------------------------------

type Database struct {
	env *mdb.Env
	// In this package, a DBI is obtained only through Open/Open2, and is never closed until
	// Context.Close(), in which all dbis are closed automatically.
	buckets map[string]mdb.DBI
}

type Stat mdb.Stat
type Info mdb.Info

func Version() string {
	return mdb.Version()
}

func Open(path string, buckets []string) (*Database, error) {
	return Open2(path, buckets, MAP_SIZE_DEFAULT)
}

func Open2(path string, buckets []string, maxMapSize uint64) (db *Database, err error) {
	// TODO (Potential bug):
	// From mdb_env_open's doc,
	//  "If this function fails, #mdb_env_close() must be called to discard the #MDB_env handle."
	// But mdb.NewEnv doesnot call mdb_env_close() when it fails, AND it just return nil as env.
	// Patch gomdb if this turns out to be a big issue.
	env, err := mdb.NewEnv()
	defer func() {
		if err != nil && env != nil {
			env.Close()
			db.env = nil // TODO: Bug - db could be nil
		}
	}()
	if err != nil {
		return
	}

	err = env.SetMapSize(maxMapSize)
	if err != nil {
		return
	}

	// http://www.openldap.org/lists/openldap-technical/201305/msg00176.html
	err = env.SetMaxDBs((mdb.DBI)(len(buckets)))
	if err != nil {
		return
	}

	MDB_NOTLS := uint(0x200000)
	MDB_NORDAHEAD := uint(0x800000)
	err = env.Open(path, MDB_NOTLS|MDB_NORDAHEAD, 0664)
	if err != nil {
		return
	}

	bucketCache := make(map[string]mdb.DBI)
	db = &Database{env, nil}

	err = db.TransactionalRW(func(txn *ReadWriteTxn) error {
		for _, name := range buckets {
			if name == "" {
				return errors.New("Bucket name is empty")
			}
			dbi, err := txn.txn.DBIOpen(&name, mdb.CREATE)
			if err != nil {
				return err
			} else {
				bucketCache[name] = dbi
			}
		}
		return nil
	})

	if err != nil {
		return
	} else {
		db.buckets = bucketCache
	}

	return
}

func (db *Database) Close() {
	if db.env != nil {
		db.env.Close() // all opened dbis are closed during this process
	}
}

func (db *Database) Stat() *Stat {
	stat, err := db.env.Stat()
	if err != nil { // Possible errors: EINVAL
		panic(err)
	}
	return (*Stat)(stat)
}

func (db *Database) Info() *Info {
	info, err := db.env.Info()
	if err != nil { // error when env == nil, so panic
		panic(err)
	}
	return (*Info)(info)
}

func (db *Database) TransactionalR(f func(ReadTxner)) {
	txn, err := db.env.BeginTxn(nil, mdb.RDONLY)
	if err != nil { // Possible Errors: MDB_PANIC, MDB_MAP_RESIZED, MDB_READERS_FULL, ENOMEM
		panic(err)
	}

	var panicF interface{} // panic from f
	rdTxn := ReadTxn{db.buckets, txn, nil}

	defer func() {
		for _, itr := range rdTxn.itrs {
			itr.Close() // no panic
		}
		rdTxn.itrs = nil

		txn.Abort()
		if panicF != nil {
			panic(panicF) // re-panic
		}
	}()

	func() {
		defer func() {
			panicF = recover()
		}()
		f(&rdTxn)
	}()
}

func (db *Database) TransactionalRW(f func(*ReadWriteTxn) error) (err error) {
	txn, err := db.env.BeginTxn(nil, 0)
	if err != nil { // Possible Errors: MDB_PANIC, MDB_MAP_RESIZED, MDB_READERS_FULL, ENOMEM
		panic(err)
	}

	var panicF interface{} // panic from f
	rwCtx := ReadWriteTxn{db.env, &ReadTxn{db.buckets, txn, nil}}

	defer func() {
		for _, itr := range rwCtx.itrs {
			itr.Close() // no panic
		}
		rwCtx.itrs = nil

		if err == nil && panicF == nil {
			e := txn.Commit()
			if e != nil { // Possible errors: EINVAL, ENOSPEC, EIO, ENOMEM
				panic(e)
			}
		} else {
			txn.Abort()
			if panicF != nil {
				panic(panicF) // re-panic
			}
		}
	}()

	func() {
		defer func() {
			panicF = recover()
		}()
		err = f(&rwCtx)
	}()

	return
}
<commit_msg>Add an error log.<commit_after>package lmdb

import (
	"errors"
	mdb "github.com/szferi/gomdb"
	"log"
)

// Thread Safety
// 1) NOTLS mode is used exclusively, which allows read txns to freely migrate across
//    threads and for a single thread to maintain multiple read txns. This enables mostly
//    care-free use of read txns.
// 2) Most objects can be safely called by a single caller from a single thread, and usually it
//    only makes sense to have a single caller, except in the case of Database.
// 3) Most Database methods are thread-safe, and may be called concurrently, except for
//    Database.Close().
// 4) A write txn may only be used from the thread it was created on.
// 5) A read-only txn can move across threads, but it cannot be used concurrently from multiple
//    threads.
// 6) Iterator is not thread-safe, but it does not make sense to use it on any thread except the
//    thread that currently owns its associated txn.
//
//----------------------------------------------------------------------------------------
//
// Best practice:
// 1) Use iterators only in the txn that they are created
// 2) DO NOT modify the memory slice from GetNoCopy
// 3) Make sure all read/write txns are finished before Database.Close().

const (
	// There is no penalty for making this huge.
	// If you are on a 32-bit system, use Open2 and specify a smaller map size.
	MAP_SIZE_DEFAULT uint64 = 64 * 1024 * 1024 * 1024 * 1024 // 64TB
)

type RWTxnCreator interface {
	TransactionalRW(func(*ReadWriteTxn) error) error
}

type dryRunDummyError struct{}

func (d dryRunDummyError) Error() string {
	return "Dummy error of dry running a db transaction."
}

// Dry run the db transaction, i.e. always rollback, even if the returned error is nil.
func DryRunRWTransaction(rwtxer RWTxnCreator, f func(*ReadWriteTxn) error) error {
	err := rwtxer.TransactionalRW(func(rwtx *ReadWriteTxn) error {
		err := f(rwtx)
		if err == nil {
			err = dryRunDummyError{}
		}
		return err
	})

	if _, ok := err.(dryRunDummyError); ok {
		err = nil
	}
	return err
}

//--------------------------------- Database ---------------------------------------------

type Database struct {
	env *mdb.Env
	// In this package, a DBI is obtained only through Open/Open2, and is never closed until
	// Context.Close(), in which all dbis are closed automatically.
	buckets map[string]mdb.DBI
}

type Stat mdb.Stat
type Info mdb.Info

func Version() string {
	return mdb.Version()
}

func Open(path string, buckets []string) (*Database, error) {
	return Open2(path, buckets, MAP_SIZE_DEFAULT)
}

func Open2(path string, buckets []string, maxMapSize uint64) (db *Database, err error) {
	// TODO (Potential bug):
	// From mdb_env_open's doc,
	//  "If this function fails, #mdb_env_close() must be called to discard the #MDB_env handle."
	// But mdb.NewEnv doesnot call mdb_env_close() when it fails, AND it just return nil as env.
	// Patch gomdb if this turns out to be a big issue.
	env, err := mdb.NewEnv()
	defer func() {
		if err != nil && env != nil {
			log.Printf("[ERROR] Open db failed. %v", err)
			env.Close()
			db.env = nil // TODO: Bug - db could be nil
		}
	}()
	if err != nil {
		return
	}

	err = env.SetMapSize(maxMapSize)
	if err != nil {
		return
	}

	// http://www.openldap.org/lists/openldap-technical/201305/msg00176.html
	err = env.SetMaxDBs((mdb.DBI)(len(buckets)))
	if err != nil {
		return
	}

	MDB_NOTLS := uint(0x200000)
	MDB_NORDAHEAD := uint(0x800000)
	err = env.Open(path, MDB_NOTLS|MDB_NORDAHEAD, 0664)
	if err != nil {
		return
	}

	bucketCache := make(map[string]mdb.DBI)
	db = &Database{env, nil}

	err = db.TransactionalRW(func(txn *ReadWriteTxn) error {
		for _, name := range buckets {
			if name == "" {
				return errors.New("Bucket name is empty")
			}
			dbi, err := txn.txn.DBIOpen(&name, mdb.CREATE)
			if err != nil {
				return err
			} else {
				bucketCache[name] = dbi
			}
		}
		return nil
	})

	if err != nil {
		return
	} else {
		db.buckets = bucketCache
	}

	return
}

func (db *Database) Close() {
	if db.env != nil {
		db.env.Close() // all opened dbis are closed during this process
	}
}

func (db *Database) Stat() *Stat {
	stat, err := db.env.Stat()
	if err != nil { // Possible errors: EINVAL
		panic(err)
	}
	return (*Stat)(stat)
}

func (db *Database) Info() *Info {
	info, err := db.env.Info()
	if err != nil { // error when env == nil, so panic
		panic(err)
	}
	return (*Info)(info)
}

func (db *Database) TransactionalR(f func(ReadTxner)) {
	txn, err := db.env.BeginTxn(nil, mdb.RDONLY)
	if err != nil { // Possible Errors: MDB_PANIC, MDB_MAP_RESIZED, MDB_READERS_FULL, ENOMEM
		panic(err)
	}

	var panicF interface{} // panic from f
	rdTxn := ReadTxn{db.buckets, txn, nil}

	defer func() {
		for _, itr := range rdTxn.itrs {
			itr.Close() // no panic
		}
		rdTxn.itrs = nil

		txn.Abort()
		if panicF != nil {
			panic(panicF) // re-panic
		}
	}()

	func() {
		defer func() {
			panicF = recover()
		}()
		f(&rdTxn)
	}()
}

func (db *Database) TransactionalRW(f func(*ReadWriteTxn) error) (err error) {
	txn, err := db.env.BeginTxn(nil, 0)
	if err != nil { // Possible Errors: MDB_PANIC, MDB_MAP_RESIZED, MDB_READERS_FULL, ENOMEM
		panic(err)
	}

	var panicF interface{} // panic from f
	rwCtx := ReadWriteTxn{db.env, &ReadTxn{db.buckets, txn, nil}}

	defer func() {
		for _, itr := range rwCtx.itrs {
			itr.Close() // no panic
		}
		rwCtx.itrs = nil

		if err == nil && panicF == nil {
			e := txn.Commit()
			if e != nil { // Possible errors: EINVAL, ENOSPEC, EIO, ENOMEM
				panic(e)
			}
		} else {
			txn.Abort()
			if panicF != nil {
				panic(panicF) // re-panic
			}
		}
	}()

	func() {
		defer func() {
			panicF = recover()
		}()
		err = f(&rwCtx)
	}()

	return
}
<|endoftext|>


<commit_before>// Copyright 2009 The GoGo Authors. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.

//
// GoGo library functions
//

package libgogo

func StringAppend(str *string, char byte) { //TODO (SC): Get rid of magic string concatenation (needs memory management!) and cast operator
  *str += string(char);
}

func Min(a uint64, b uint64) uint64 {
    var result uint64 = b;
    if a < b {
        result = a;
    }
    return result;
}

func StringLength(str string) uint64;

func StringCompare(str1 string, str2 string) uint64 {
    var i uint64;
    var equal uint64 = 1;
    var strlen1 uint64 = StringLength(str1);
    var strlen2 uint64 = StringLength(str2);
    if strlen1 != strlen2 {
       equal = 0;
    } else {
        for i = 0; i < strlen1; i = i +1 {
            if str1[i] != str2[i] {
                equal = 0;
            }
        }
    }
    return equal;
}

func ToIntFromByte(b byte) uint64;

func ToByteFromInt(i uint64) byte;

func StringToInt(str string) uint64 {
    var n uint64 = StringLength(str);
    var i uint64;
    var val uint64 = 0;
    for i = 0; i < n ; i = i +1 {
        val = val * 10;
        val = val + ToIntFromByte(str[i]) - 48;
    }
    return val;
}

func Exit(code uint64);

func ExitError(msg string, code uint64) {
    PrintString(msg);
    PrintChar('\n');
    Exit(code);
}

func Write(fd uint64, text string, length uint64) uint64;

func PrintString(msg string) {
    Write(1, msg, StringLength(msg));
}

func PrintChar(char byte);

func PrintNumber(num uint64) {
    var i uint64;
    var buf [255]byte;
    for i = 0; num != 0; i = i +1 {
        buf[i] = ToByteFromInt(num - (num / 10) * 10 + 48);
        num = num / 10;
    }
    if i == 0 { //Special case: 0
        buf[0] = 48;
        i = 1;
    }
    for ; i != 0; i = i -1 {
        PrintChar(buf[i]);
    }
    PrintChar(buf[0]);
}

func Read(fd uint64, buffer string, buffer_size uint64) uint64;

func GetChar(fd uint64) byte;

func FileOpen(filename string, flags uint64) uint64;

func FileClose(fd uint64) uint64;
<commit_msg>libgogo.go: Fix StringCompare to return 0 for success<commit_after>// Copyright 2009 The GoGo Authors. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.

//
// GoGo library functions
//

package libgogo

func StringAppend(str *string, char byte) { //TODO (SC): Get rid of magic string concatenation (needs memory management!) and cast operator
  *str += string(char);
}

func Min(a uint64, b uint64) uint64 {
    var result uint64 = b;
    if a < b {
        result = a;
    }
    return result;
}

func StringLength(str string) uint64;

func StringCompare(str1 string, str2 string) uint64 {
    var i uint64;
    var equal uint64 = 0;
    var strlen1 uint64 = StringLength(str1);
    var strlen2 uint64 = StringLength(str2);
    if strlen1 != strlen2 {
       equal = 1;
    } else {
        for i = 0; i < strlen1; i = i +1 {
            if str1[i] != str2[i] {
                equal = 1;
            }
        }
    }
    return equal;
}

func ToIntFromByte(b byte) uint64;

func ToByteFromInt(i uint64) byte;

func StringToInt(str string) uint64 {
    var n uint64 = StringLength(str);
    var i uint64;
    var val uint64 = 0;
    for i = 0; i < n ; i = i +1 {
        val = val * 10;
        val = val + ToIntFromByte(str[i]) - 48;
    }
    return val;
}

func Exit(code uint64);

func ExitError(msg string, code uint64) {
    PrintString(msg);
    PrintChar('\n');
    Exit(code);
}

func Write(fd uint64, text string, length uint64) uint64;

func PrintString(msg string) {
    Write(1, msg, StringLength(msg));
}

func PrintChar(char byte);

func PrintNumber(num uint64) {
    var i uint64;
    var buf [255]byte;
    for i = 0; num != 0; i = i +1 {
        buf[i] = ToByteFromInt(num - (num / 10) * 10 + 48);
        num = num / 10;
    }
    if i == 0 { //Special case: 0
        buf[0] = 48;
        i = 1;
    }
    for ; i != 0; i = i -1 {
        PrintChar(buf[i]);
    }
    PrintChar(buf[0]);
}

func Read(fd uint64, buffer string, buffer_size uint64) uint64;

func GetChar(fd uint64) byte;

func FileOpen(filename string, flags uint64) uint64;

func FileClose(fd uint64) uint64;
<|endoftext|>


<commit_before>package fakes

import (
	"bytes"
	"errors"
	"fmt"
	"os"
	"path/filepath"
	"strings"
	"sync"

	gouuid "github.com/nu7hatch/gouuid"

	bosherr "github.com/cloudfoundry/bosh-agent/errors"
	boshsys "github.com/cloudfoundry/bosh-agent/system"
)

type FakeFileType string

const (
	FakeFileTypeFile    FakeFileType = "file"
	FakeFileTypeSymlink FakeFileType = "symlink"
	FakeFileTypeDir     FakeFileType = "dir"
)

type FakeFileSystem struct {
	files     map[string]*FakeFileStats
	filesLock sync.Mutex

	HomeDirUsername string
	HomeDirHomePath string

	openFiles   map[string]*FakeFile
	OpenFileErr error

	ReadFileError    error
	WriteToFileError error
	SymlinkError     error

	MkdirAllError       error
	mkdirAllErrorByPath map[string]error

	ChownErr error
	ChmodErr error

	CopyFileError error

	RenameError    error
	RenameOldPaths []string
	RenameNewPaths []string

	RemoveAllError       error
	removeAllErrorByPath map[string]error

	ReadLinkError error

	TempFileError  error
	ReturnTempFile *os.File

	TempDirDir   string
	TempDirError error

	GlobErr  error
	globsMap map[string][][]string
}

type FakeFileStats struct {
	FileType FakeFileType

	FileMode os.FileMode
	Username string

	SymlinkTarget string

	Content []byte
}

func (stats FakeFileStats) StringContents() string {
	return string(stats.Content)
}

type FakeFileInfo struct {
	os.FileInfo
	file FakeFile
}

func (fi FakeFileInfo) Size() int64 {
	return int64(len(fi.file.Contents))
}

type FakeFile struct {
	path string
	fs   *FakeFileSystem

	WriteErr error
	Contents []byte

	ReadErr   error
	ReadAtErr error

	CloseErr error

	StatErr error
}

func NewFakeFile(fs *FakeFileSystem) *FakeFile {
	return &FakeFile{fs: fs}
}

func (f *FakeFile) Write(contents []byte) (int, error) {
	if f.WriteErr != nil {
		return 0, f.WriteErr
	}

	f.fs.filesLock.Lock()
	defer f.fs.filesLock.Unlock()

	stats := f.fs.getOrCreateFile(f.path)
	stats.Content = contents

	f.Contents = contents
	return len(contents), nil
}

func (f *FakeFile) Read(p []byte) (int, error) {
	copy(p, f.Contents)
	return len(f.Contents), f.ReadErr
}

func (f *FakeFile) ReadAt(b []byte, offset int64) (int, error) {
	copy(b, f.Contents[offset:])
	return len(f.Contents[offset:]), f.ReadAtErr
}

func (f *FakeFile) Close() error {
	return f.CloseErr
}

func (f FakeFile) Stat() (os.FileInfo, error) {
	return FakeFileInfo{file: f}, f.StatErr
}

func NewFakeFileSystem() *FakeFileSystem {
	return &FakeFileSystem{
		files:                map[string]*FakeFileStats{},
		openFiles:            map[string]*FakeFile{},
		globsMap:             map[string][][]string{},
		removeAllErrorByPath: map[string]error{},
		mkdirAllErrorByPath:  map[string]error{},
	}
}

func (fs *FakeFileSystem) GetFileTestStat(path string) *FakeFileStats {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	return fs.files[path]
}

func (fs *FakeFileSystem) HomeDir(username string) (string, error) {
	fs.HomeDirUsername = username
	return fs.HomeDirHomePath, nil
}

func (fs *FakeFileSystem) RegisterMkdirAllError(path string, err error) {
	if _, ok := fs.mkdirAllErrorByPath[path]; ok {
		panic(fmt.Sprintf("MkdirAll error is already set for path: %s", path))
	}
	fs.mkdirAllErrorByPath[path] = err
}

func (fs *FakeFileSystem) MkdirAll(path string, perm os.FileMode) error {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.MkdirAllError != nil {
		return fs.MkdirAllError
	}

	if fs.mkdirAllErrorByPath[path] != nil {
		return fs.mkdirAllErrorByPath[path]
	}

	stats := fs.getOrCreateFile(path)
	stats.FileMode = perm
	stats.FileType = FakeFileTypeDir
	return nil
}

func (fs *FakeFileSystem) RegisterOpenFile(path string, file *FakeFile) {
	fs.openFiles[path] = file
}

func (fs *FakeFileSystem) OpenFile(path string, flag int, perm os.FileMode) (boshsys.ReadWriteCloseStater, error) {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.OpenFileErr != nil {
		return nil, fs.OpenFileErr
	}

	// Make sure to record a reference for FileExist, etc. to work
	stats := fs.getOrCreateFile(path)
	stats.FileMode = perm
	stats.FileType = FakeFileTypeFile

	if fs.openFiles[path] != nil {
		return fs.openFiles[path], nil
	}

	file := &FakeFile{
		path: path,
		fs:   fs,
	}

	return file, nil
}

func (fs *FakeFileSystem) Chown(path, username string) error {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	// check early to avoid requiring file presence
	if fs.ChownErr != nil {
		return fs.ChownErr
	}

	stats := fs.files[path]
	if stats == nil {
		return fmt.Errorf("Path does not exist: %s", path)
	}

	stats.Username = username
	return nil
}

func (fs *FakeFileSystem) Chmod(path string, perm os.FileMode) error {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	// check early to avoid requiring file presence
	if fs.ChmodErr != nil {
		return fs.ChmodErr
	}

	stats := fs.files[path]
	if stats == nil {
		return fmt.Errorf("Path does not exist: %s", path)
	}

	stats.FileMode = perm
	return nil
}

func (fs *FakeFileSystem) WriteFileString(path, content string) (err error) {
	return fs.WriteFile(path, []byte(content))
}

func (fs *FakeFileSystem) WriteFile(path string, content []byte) (err error) {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.WriteToFileError != nil {
		return fs.WriteToFileError
	}

	stats := fs.getOrCreateFile(path)
	stats.FileType = FakeFileTypeFile
	stats.Content = content
	return nil
}

func (fs *FakeFileSystem) ConvergeFileContents(path string, content []byte) (bool, error) {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.WriteToFileError != nil {
		return false, fs.WriteToFileError
	}

	stats := fs.getOrCreateFile(path)
	stats.FileType = FakeFileTypeFile

	if bytes.Compare(stats.Content, content) != 0 {
		stats.Content = content
		return true, nil
	}

	return false, nil
}

func (fs *FakeFileSystem) ReadFileString(path string) (string, error) {
	bytes, err := fs.ReadFile(path)
	if err != nil {
		return "", err
	}

	return string(bytes), nil
}

func (fs *FakeFileSystem) ReadFile(path string) ([]byte, error) {
	stats := fs.GetFileTestStat(path)
	if stats != nil {
		if fs.ReadFileError != nil {
			return nil, fs.ReadFileError
		}
		return stats.Content, nil
	}
	return nil, errors.New("File not found")
}

func (fs *FakeFileSystem) FileExists(path string) bool {
	return fs.GetFileTestStat(path) != nil
}

func (fs *FakeFileSystem) Rename(oldPath, newPath string) error {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.RenameError != nil {
		return fs.RenameError
	}

	if fs.files[filepath.Dir(newPath)] == nil {
		return errors.New("Parent directory does not exist")
	}

	stats := fs.files[oldPath]
	if stats == nil {
		return errors.New("Old path did not exist")
	}

	fs.RenameOldPaths = append(fs.RenameOldPaths, oldPath)
	fs.RenameNewPaths = append(fs.RenameNewPaths, newPath)

	newStats := fs.getOrCreateFile(newPath)
	newStats.Content = stats.Content
	newStats.FileMode = stats.FileMode
	newStats.FileType = stats.FileType

	// Ignore error from RemoveAll
	fs.removeAll(oldPath)

	return nil
}

func (fs *FakeFileSystem) Symlink(oldPath, newPath string) (err error) {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.SymlinkError == nil {
		stats := fs.getOrCreateFile(newPath)
		stats.FileType = FakeFileTypeSymlink
		stats.SymlinkTarget = oldPath
		return
	}

	err = fs.SymlinkError
	return
}

func (fs *FakeFileSystem) ReadLink(symlinkPath string) (string, error) {
	if fs.ReadLinkError != nil {
		return "", fs.ReadLinkError
	}

	stat := fs.GetFileTestStat(symlinkPath)
	if stat != nil {
		return stat.SymlinkTarget, nil
	}

	return "", os.ErrNotExist
}

func (fs *FakeFileSystem) CopyFile(srcPath, dstPath string) error {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.CopyFileError != nil {
		return fs.CopyFileError
	}

	fs.files[dstPath] = fs.files[srcPath]
	return nil
}

func (fs *FakeFileSystem) TempFile(prefix string) (file *os.File, err error) {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.TempFileError != nil {
		return nil, fs.TempFileError
	}

	if fs.ReturnTempFile != nil {
		return fs.ReturnTempFile, nil
	}

	file, err = os.Open("/dev/null")
	if err != nil {
		err = bosherr.WrapError(err, "Opening /dev/null")
		return
	}

	// Make sure to record a reference for FileExist, etc. to work
	stats := fs.getOrCreateFile(file.Name())
	stats.FileType = FakeFileTypeFile
	return
}

func (fs *FakeFileSystem) TempDir(prefix string) (string, error) {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.TempDirError != nil {
		return "", fs.TempDirError
	}

	var path string
	if len(fs.TempDirDir) > 0 {
		path = fs.TempDirDir
	} else {
		uuid, err := gouuid.NewV4()
		if err != nil {
			return "", err
		}

		path = uuid.String()
	}

	// Make sure to record a reference for FileExist, etc. to work
	stats := fs.getOrCreateFile(path)
	stats.FileType = FakeFileTypeDir

	return path, nil
}

func (fs *FakeFileSystem) RegisterRemoveAllError(path string, err error) {
	if _, ok := fs.removeAllErrorByPath[path]; ok {
		panic(fmt.Sprintf("RemoveAll error is already set for path: %s", path))
	}
	fs.removeAllErrorByPath[path] = err
}

func (fs *FakeFileSystem) RemoveAll(path string) error {
	if path == "" {
		panic("RemoveAll requires path")
	}

	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.RemoveAllError != nil {
		return fs.RemoveAllError
	}

	if fs.removeAllErrorByPath[path] != nil {
		return fs.removeAllErrorByPath[path]
	}

	return fs.removeAll(path)
}

func (fs *FakeFileSystem) removeAll(path string) error {
	filesToRemove := []string{}

	for name := range fs.files {
		if strings.HasPrefix(name, path) {
			filesToRemove = append(filesToRemove, name)
		}
	}

	for _, name := range filesToRemove {
		delete(fs.files, name)
	}
	return nil
}

func (fs *FakeFileSystem) Glob(pattern string) (matches []string, err error) {
	remainingMatches, found := fs.globsMap[pattern]
	if found {
		matches = remainingMatches[0]
		if len(remainingMatches) > 1 {
			fs.globsMap[pattern] = remainingMatches[1:]
		}
	} else {
		matches = []string{}
	}
	return matches, fs.GlobErr
}

func (fs *FakeFileSystem) SetGlob(pattern string, matches ...[]string) {
	fs.globsMap[pattern] = matches
}

func (fs *FakeFileSystem) getOrCreateFile(path string) *FakeFileStats {
	stats := fs.files[path]
	if stats == nil {
		stats = new(FakeFileStats)
		fs.files[path] = stats
	}
	return stats
}
<commit_msg>Remember stubbed FakeFileSystem TempFile if set<commit_after>package fakes

import (
	"bytes"
	"errors"
	"fmt"
	"os"
	"path/filepath"
	"strings"
	"sync"

	gouuid "github.com/nu7hatch/gouuid"

	bosherr "github.com/cloudfoundry/bosh-agent/errors"
	boshsys "github.com/cloudfoundry/bosh-agent/system"
)

type FakeFileType string

const (
	FakeFileTypeFile    FakeFileType = "file"
	FakeFileTypeSymlink FakeFileType = "symlink"
	FakeFileTypeDir     FakeFileType = "dir"
)

type FakeFileSystem struct {
	files     map[string]*FakeFileStats
	filesLock sync.Mutex

	HomeDirUsername string
	HomeDirHomePath string

	openFiles   map[string]*FakeFile
	OpenFileErr error

	ReadFileError    error
	WriteToFileError error
	SymlinkError     error

	MkdirAllError       error
	mkdirAllErrorByPath map[string]error

	ChownErr error
	ChmodErr error

	CopyFileError error

	RenameError    error
	RenameOldPaths []string
	RenameNewPaths []string

	RemoveAllError       error
	removeAllErrorByPath map[string]error

	ReadLinkError error

	TempFileError  error
	ReturnTempFile *os.File

	TempDirDir   string
	TempDirError error

	GlobErr  error
	globsMap map[string][][]string
}

type FakeFileStats struct {
	FileType FakeFileType

	FileMode os.FileMode
	Username string

	SymlinkTarget string

	Content []byte
}

func (stats FakeFileStats) StringContents() string {
	return string(stats.Content)
}

type FakeFileInfo struct {
	os.FileInfo
	file FakeFile
}

func (fi FakeFileInfo) Size() int64 {
	return int64(len(fi.file.Contents))
}

type FakeFile struct {
	path string
	fs   *FakeFileSystem

	WriteErr error
	Contents []byte

	ReadErr   error
	ReadAtErr error

	CloseErr error

	StatErr error
}

func NewFakeFile(fs *FakeFileSystem) *FakeFile {
	return &FakeFile{fs: fs}
}

func (f *FakeFile) Write(contents []byte) (int, error) {
	if f.WriteErr != nil {
		return 0, f.WriteErr
	}

	f.fs.filesLock.Lock()
	defer f.fs.filesLock.Unlock()

	stats := f.fs.getOrCreateFile(f.path)
	stats.Content = contents

	f.Contents = contents
	return len(contents), nil
}

func (f *FakeFile) Read(p []byte) (int, error) {
	copy(p, f.Contents)
	return len(f.Contents), f.ReadErr
}

func (f *FakeFile) ReadAt(b []byte, offset int64) (int, error) {
	copy(b, f.Contents[offset:])
	return len(f.Contents[offset:]), f.ReadAtErr
}

func (f *FakeFile) Close() error {
	return f.CloseErr
}

func (f FakeFile) Stat() (os.FileInfo, error) {
	return FakeFileInfo{file: f}, f.StatErr
}

func NewFakeFileSystem() *FakeFileSystem {
	return &FakeFileSystem{
		files:                map[string]*FakeFileStats{},
		openFiles:            map[string]*FakeFile{},
		globsMap:             map[string][][]string{},
		removeAllErrorByPath: map[string]error{},
		mkdirAllErrorByPath:  map[string]error{},
	}
}

func (fs *FakeFileSystem) GetFileTestStat(path string) *FakeFileStats {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	return fs.files[path]
}

func (fs *FakeFileSystem) HomeDir(username string) (string, error) {
	fs.HomeDirUsername = username
	return fs.HomeDirHomePath, nil
}

func (fs *FakeFileSystem) RegisterMkdirAllError(path string, err error) {
	if _, ok := fs.mkdirAllErrorByPath[path]; ok {
		panic(fmt.Sprintf("MkdirAll error is already set for path: %s", path))
	}
	fs.mkdirAllErrorByPath[path] = err
}

func (fs *FakeFileSystem) MkdirAll(path string, perm os.FileMode) error {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.MkdirAllError != nil {
		return fs.MkdirAllError
	}

	if fs.mkdirAllErrorByPath[path] != nil {
		return fs.mkdirAllErrorByPath[path]
	}

	stats := fs.getOrCreateFile(path)
	stats.FileMode = perm
	stats.FileType = FakeFileTypeDir
	return nil
}

func (fs *FakeFileSystem) RegisterOpenFile(path string, file *FakeFile) {
	fs.openFiles[path] = file
}

func (fs *FakeFileSystem) OpenFile(path string, flag int, perm os.FileMode) (boshsys.ReadWriteCloseStater, error) {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.OpenFileErr != nil {
		return nil, fs.OpenFileErr
	}

	// Make sure to record a reference for FileExist, etc. to work
	stats := fs.getOrCreateFile(path)
	stats.FileMode = perm
	stats.FileType = FakeFileTypeFile

	if fs.openFiles[path] != nil {
		return fs.openFiles[path], nil
	}

	file := &FakeFile{
		path: path,
		fs:   fs,
	}

	return file, nil
}

func (fs *FakeFileSystem) Chown(path, username string) error {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	// check early to avoid requiring file presence
	if fs.ChownErr != nil {
		return fs.ChownErr
	}

	stats := fs.files[path]
	if stats == nil {
		return fmt.Errorf("Path does not exist: %s", path)
	}

	stats.Username = username
	return nil
}

func (fs *FakeFileSystem) Chmod(path string, perm os.FileMode) error {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	// check early to avoid requiring file presence
	if fs.ChmodErr != nil {
		return fs.ChmodErr
	}

	stats := fs.files[path]
	if stats == nil {
		return fmt.Errorf("Path does not exist: %s", path)
	}

	stats.FileMode = perm
	return nil
}

func (fs *FakeFileSystem) WriteFileString(path, content string) (err error) {
	return fs.WriteFile(path, []byte(content))
}

func (fs *FakeFileSystem) WriteFile(path string, content []byte) (err error) {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.WriteToFileError != nil {
		return fs.WriteToFileError
	}

	stats := fs.getOrCreateFile(path)
	stats.FileType = FakeFileTypeFile
	stats.Content = content
	return nil
}

func (fs *FakeFileSystem) ConvergeFileContents(path string, content []byte) (bool, error) {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.WriteToFileError != nil {
		return false, fs.WriteToFileError
	}

	stats := fs.getOrCreateFile(path)
	stats.FileType = FakeFileTypeFile

	if bytes.Compare(stats.Content, content) != 0 {
		stats.Content = content
		return true, nil
	}

	return false, nil
}

func (fs *FakeFileSystem) ReadFileString(path string) (string, error) {
	bytes, err := fs.ReadFile(path)
	if err != nil {
		return "", err
	}

	return string(bytes), nil
}

func (fs *FakeFileSystem) ReadFile(path string) ([]byte, error) {
	stats := fs.GetFileTestStat(path)
	if stats != nil {
		if fs.ReadFileError != nil {
			return nil, fs.ReadFileError
		}
		return stats.Content, nil
	}
	return nil, errors.New("File not found")
}

func (fs *FakeFileSystem) FileExists(path string) bool {
	return fs.GetFileTestStat(path) != nil
}

func (fs *FakeFileSystem) Rename(oldPath, newPath string) error {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.RenameError != nil {
		return fs.RenameError
	}

	if fs.files[filepath.Dir(newPath)] == nil {
		return errors.New("Parent directory does not exist")
	}

	stats := fs.files[oldPath]
	if stats == nil {
		return errors.New("Old path did not exist")
	}

	fs.RenameOldPaths = append(fs.RenameOldPaths, oldPath)
	fs.RenameNewPaths = append(fs.RenameNewPaths, newPath)

	newStats := fs.getOrCreateFile(newPath)
	newStats.Content = stats.Content
	newStats.FileMode = stats.FileMode
	newStats.FileType = stats.FileType

	// Ignore error from RemoveAll
	fs.removeAll(oldPath)

	return nil
}

func (fs *FakeFileSystem) Symlink(oldPath, newPath string) (err error) {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.SymlinkError == nil {
		stats := fs.getOrCreateFile(newPath)
		stats.FileType = FakeFileTypeSymlink
		stats.SymlinkTarget = oldPath
		return
	}

	err = fs.SymlinkError
	return
}

func (fs *FakeFileSystem) ReadLink(symlinkPath string) (string, error) {
	if fs.ReadLinkError != nil {
		return "", fs.ReadLinkError
	}

	stat := fs.GetFileTestStat(symlinkPath)
	if stat != nil {
		return stat.SymlinkTarget, nil
	}

	return "", os.ErrNotExist
}

func (fs *FakeFileSystem) CopyFile(srcPath, dstPath string) error {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.CopyFileError != nil {
		return fs.CopyFileError
	}

	fs.files[dstPath] = fs.files[srcPath]
	return nil
}

func (fs *FakeFileSystem) TempFile(prefix string) (file *os.File, err error) {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.TempFileError != nil {
		return nil, fs.TempFileError
	}

	if fs.ReturnTempFile != nil {
		file = fs.ReturnTempFile
	} else {
		file, err = os.Open("/dev/null")
		if err != nil {
			err = bosherr.WrapError(err, "Opening /dev/null")
			return
		}
	}

	// Make sure to record a reference for FileExist, etc. to work
	stats := fs.getOrCreateFile(file.Name())
	stats.FileType = FakeFileTypeFile
	return
}

func (fs *FakeFileSystem) TempDir(prefix string) (string, error) {
	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.TempDirError != nil {
		return "", fs.TempDirError
	}

	var path string
	if len(fs.TempDirDir) > 0 {
		path = fs.TempDirDir
	} else {
		uuid, err := gouuid.NewV4()
		if err != nil {
			return "", err
		}

		path = uuid.String()
	}

	// Make sure to record a reference for FileExist, etc. to work
	stats := fs.getOrCreateFile(path)
	stats.FileType = FakeFileTypeDir

	return path, nil
}

func (fs *FakeFileSystem) RegisterRemoveAllError(path string, err error) {
	if _, ok := fs.removeAllErrorByPath[path]; ok {
		panic(fmt.Sprintf("RemoveAll error is already set for path: %s", path))
	}
	fs.removeAllErrorByPath[path] = err
}

func (fs *FakeFileSystem) RemoveAll(path string) error {
	if path == "" {
		panic("RemoveAll requires path")
	}

	fs.filesLock.Lock()
	defer fs.filesLock.Unlock()

	if fs.RemoveAllError != nil {
		return fs.RemoveAllError
	}

	if fs.removeAllErrorByPath[path] != nil {
		return fs.removeAllErrorByPath[path]
	}

	return fs.removeAll(path)
}

func (fs *FakeFileSystem) removeAll(path string) error {
	filesToRemove := []string{}

	for name := range fs.files {
		if strings.HasPrefix(name, path) {
			filesToRemove = append(filesToRemove, name)
		}
	}

	for _, name := range filesToRemove {
		delete(fs.files, name)
	}
	return nil
}

func (fs *FakeFileSystem) Glob(pattern string) (matches []string, err error) {
	remainingMatches, found := fs.globsMap[pattern]
	if found {
		matches = remainingMatches[0]
		if len(remainingMatches) > 1 {
			fs.globsMap[pattern] = remainingMatches[1:]
		}
	} else {
		matches = []string{}
	}
	return matches, fs.GlobErr
}

func (fs *FakeFileSystem) SetGlob(pattern string, matches ...[]string) {
	fs.globsMap[pattern] = matches
}

func (fs *FakeFileSystem) getOrCreateFile(path string) *FakeFileStats {
	stats := fs.files[path]
	if stats == nil {
		stats = new(FakeFileStats)
		fs.files[path] = stats
	}
	return stats
}
<|endoftext|>


<commit_before>//WARNING , this Package should not use other Packages to prevent cycling includes. And it should not need to include others

package Flags

import (
	"flag"
	"path/filepath"
)

var (
	verbosity = flag.Int("v", 3, "Verbosity (0=Errors ,3=All)")
	workDir = flag.String("workdir", standardizePath("datastorage"), "Folder in which the Server stores the Userdatabase aswell as the Users uploaded data")
	port = flag.Int("port",8080,"Port for the Webserver to listen to (Port 1-1024 need SuperUser Privileges on Unix systems!)")
	TLScert = flag.String("tlscert", standardizePath("res/certificates/server.pem"), "")
	TLSkey = flag.String("tlskey", standardizePath("res/certificates/server.key"), "")

)

//Getters , Dereferencing to keep privacy

func GetVerbosity() int {
	return *verbosity
}

func GetWorkDir() string {
	return *workDir
}

func GetPort() int {
	return *port
}

func GetTLScert() string {
	return *TLScert
}

func GetTLSkey() string {
	return *TLSkey
}

// Init gets called before main()

func init() {
	flag.Parse()
}

//TODO Cant use Utils/Logging.go here because of cycling includes :/
func standardizePath(path string) string {

	newPath, err := filepath.Abs(path)
	if err != nil {
		println(err)
	}

	return newPath
}
<commit_msg>Added Todo<commit_after>//WARNING , this Package should not use other Packages to prevent cycling includes. And it should not need to include others

package Flags

import (
	"flag"
	"path/filepath"
)

var (
	verbosity = flag.Int("v", 3, "Verbosity (0=Errors ,3=All)")
	workDir = flag.String("workdir", standardizePath("datastorage"), "Folder in which the Server stores the Userdatabase aswell as the Users uploaded data")
	port = flag.Int("port",8080,"Port for the Webserver to listen to (Port 1-1024 need SuperUser Privileges on Unix systems!)")
	TLScert = flag.String("tlscert", standardizePath("res/certificates/server.pem"), "")
	TLSkey = flag.String("tlskey", standardizePath("res/certificates/server.key"), "")

)

//TODO Input Check
//Getters , Dereferencing to keep privacy. Input Ckack and Handling Here
func GetVerbosity() int {
	return *verbosity
}

func GetWorkDir() string {
	return *workDir
}

func GetPort() int {
	return *port
}

func GetTLScert() string {
	return *TLScert
}

func GetTLSkey() string {
	return *TLSkey
}

// Init gets called before main()

func init() {
	flag.Parse()
}

//TODO Cant use Utils/Logging.go here because of cycling includes :/
func standardizePath(path string) string {

	newPath, err := filepath.Abs(path)
	if err != nil {
		println(err)
	}

	return newPath
}
<|endoftext|>