<commit_before>package main

import (
	"html/template"
	"math/rand"
	"net/http"
	"os"
)

// these are the phrases we pick from when generating lipsum
var phrases = []string{
	"Hello, IT. Have you tried turning it off and on again?",
	"Uh... okay, well, the button on the side, is it glowing?",
	"Yeah, you need to turn it on... uh, the button turns it on.",
	"Yeah, you do know how a button works don't you? No, not on clothes.",
	"Hello, IT. Have you tried forcing an unexpected reboot?",
	"No, no there you go, no there you go. I just heard it come on.",
	"No, no, that's the music you heard when it come on.",
	"No, that's the music you hear when... I'm sorry are you from the past?",
	"See the driver hooks a function by patching the system call table, so its not safe to unload it unless another thread's about to jump in there and do its stuff, and you don't want to end up in the middle of invalid memory!",
	"Oh really? Then why don't you come down and make me then.",
	"Huh, what you think I'm afraid of you? I'm not afraid of you.",
	"You can come down here any time and I'll be waiting for you! [slams down phone] That told her!",
	"I mean, they have no respect for us up there! No respect whatsoever! We're all just drudgeons to them!",
	"Yes! If there were such a thing as a drudgeon, that is what we'd be to them.",
	"It's like they're pally-wally with us when there's a problem with their printer, but once it's fixed...",
	"They just toss us away like yesterday's jam.",
	"Yes! Yesterday's jam. That is what we are to them!... Actually, that doesn't work, as a thing, because, you know, jam lasts for ages.",
	"From today, dialing 999 won't get you the Emergency Services, and that's not the only thing that's changing!",
	"Nicer ambulances, faster response times and better looking drivers mean they're not just the Emergency Services, they're your Emergency Services.",
	"So, remember the new number! 0118 999! 88199, 9119 725! ... 3!",
	"Hello? I've had a bit of a tumble.",
	"Well that's easy to remember. 0118 999 88199 9119 725! ... 3!",
	"I don't see how they couldn't just keep it as it was. How hard is it to remember 911?",
	"You mean 999. Yes, yes, I mean 999! Yeah, I know. That's the American one, you berk!",
	"I'll put this over here, with the rest of the fire.",
	"0115... no... 0118... no... 0118 999 ... 3. Hello? Is this the emergency services? Then which country am I speaking to? Hello? Hello?",
	"Dear Sir stroke Madam, I am writing to inform you of a fire which has broken out at the premises of...",
	"Dear Sir stroke Madam. Fire, exclamation mark. Fire, exclamation mark. Help me, exclamation mark. 123 Carrendon Road. Looking forward to hearing from you. All the best, Maurice Moss.",
	"I'm a 32 year old IT-man who works in a basement. Yes, I do the whole Lonely Hearts thing!",
	"Shut up, do what I tell you, I'm not interested; these are just some of the things you'll be hearing if you answer this ad. I'm an idiot and I dont care about anyone but myself. P.S. No dogs!",
	"I'm going to murder you... You bloody woman!",
	"Might want to play a bit hard to get.",
	"We don't need no education. Yes you do. You've just used a double negative.",
	"How can you two... Don't Google the question, Moss!",
	"If anyone was ever rude to me, I used to carry their food around in my trousers. Oh my God! Before you brought it to their table? No, after! Of course, before! Why would I do it after?",
	"While he was eating, did you hear anyone laughing? Like... in the kitchen area? Yes! Yes I did, actually, yes I did. That'd be trouser food!",
	"OK. Moss, what did you have for breakfast this morning? Smarties cereal.",
	"Oh my God. I didn't even know Smarties made a cereal. They don't. It's just Smarties in a bowl with milk.",
	"I am a man, he's a man, we're men! Ok, tell me how your feeling. I feel delicate... and annoyed, and... I think I'm ugly.",
	"I've got Aunt Irma visiting. Oh, do you not like Aunt Irma? I've got an aunt like that.",
	"It's my term for my time of the month. Oh. What time of the month? The weekend?",
	"You know, it's high tide. But we're not on the coast. I'm closed for maintenance! Closed for maintenance? I've fallen to the communists! Well, they do have some strong arguments.",
	"Carrie, Moss! First scene in Carrie! Oh. Okay",
	"A gay musical, called Gay. That's quite gay. Gay musical? Aren't all musicals gay? This must be, like, the gayest musical ever.",
	"A story of a young man trying to find his sexuality in the uncaring Thatcher years. Warning: Contains scenes of graphic homoeroticism.",
	"Graphic homoeroticism? Does that mean they're going to get them out?",
	"You're not comfortable with your sexuality? Oh, I'm very comfortable with my sexuality, I just don't want to be slapped in the face with their sexuality.",
	"He's had quite an evening. Someone stole his wheelchair. Did you see who it was? Red bearded man.",
	"How long have you been disabled? Ten years? Ten years, and how did it happen? If that's not a rude question. Acid?",
	"When I started Reynholm Industries, I had just two things in my possession: a dream and 6 million pounds.",
	"Today I have a business empire the like of which the world has never seen the like of which. I hope it doesn't sound arrogant when I say that I am the greatest man in the world!",
	"Unbelievable! Some idiot disabled his firewall, meaning all the computers on Seven are teeming with viruses, plus I've just had to walk all the way down the motherfudging stairs, because the lifts are broken AGAIN!",
}

// index is the base html string for... index
var index = `
<!DOCTYPE html>

        <meta charset="utf-8">
        <title>IT Crowd Ipsum</title>

It Crowd Ipsum

Placeholder text taken from The IT Crowd

                {{range .Paragraphs}}
{{ . }}
{{ end }}
                <menu>
                <textarea id="text">
{{range .Paragraphs}}{{ . }}

{{ end }}
</textarea>
                    <button type="button" id="copy">Copy?</button>
                    Now press CMD + C / CTRL + C
                </menu>

Inspired by Bluth Ipsum. Made by Kivlor

`

// right up main street
func main() {
	// make sure we have a port
	port := os.Getenv("PORT")

	if port == "" {
		panic("unable to determine port")
	}

	http.HandleFunc("/", root)
	http.ListenAndServe(":"+port, nil)
}

// root is the handler for requests to "/"
func root(w http.ResponseWriter, r *http.Request) {
	// allocate a new html template
	tmpl, err := template.New("home").Parse(index)
	if err != nil {
		panic("unable to parse index")
	}

	// build the template data
	data := struct {
		Paragraphs []string
	}{
		Paragraphs: GenerateLipsum(5),
	}

	// execute the template data
	tmpl.Execute(w, data)
}

// GenerateLipsum will create a number of paragraphs using randome phrases
func GenerateLipsum(count int) []string {
	var lipsum []string
	var paragraph string

	// loop the paragraph count
	for i := 0; i < count; i++ {
		paragraph = ""
		// about 6 phrases makes a goo paragrpah
		for j := 0; j < 6; j++ {
			paragraph += phrases[rand.Intn(len(phrases))]
			paragraph += " "
		}

		// append our paragraph to lipsum
		lipsum = append(lipsum, paragraph)
	}

	// return lipsum
	return lipsum
}
<commit_msg>consistant use of 'it crowd'<commit_after>package main

import (
	"html/template"
	"math/rand"
	"net/http"
	"os"
)

// these are the phrases we pick from when generating lipsum
var phrases = []string{
	"Hello, IT. Have you tried turning it off and on again?",
	"Uh... okay, well, the button on the side, is it glowing?",
	"Yeah, you need to turn it on... uh, the button turns it on.",
	"Yeah, you do know how a button works don't you? No, not on clothes.",
	"Hello, IT. Have you tried forcing an unexpected reboot?",
	"No, no there you go, no there you go. I just heard it come on.",
	"No, no, that's the music you heard when it come on.",
	"No, that's the music you hear when... I'm sorry are you from the past?",
	"See the driver hooks a function by patching the system call table, so its not safe to unload it unless another thread's about to jump in there and do its stuff, and you don't want to end up in the middle of invalid memory!",
	"Oh really? Then why don't you come down and make me then.",
	"Huh, what you think I'm afraid of you? I'm not afraid of you.",
	"You can come down here any time and I'll be waiting for you! [slams down phone] That told her!",
	"I mean, they have no respect for us up there! No respect whatsoever! We're all just drudgeons to them!",
	"Yes! If there were such a thing as a drudgeon, that is what we'd be to them.",
	"It's like they're pally-wally with us when there's a problem with their printer, but once it's fixed...",
	"They just toss us away like yesterday's jam.",
	"Yes! Yesterday's jam. That is what we are to them!... Actually, that doesn't work, as a thing, because, you know, jam lasts for ages.",
	"From today, dialing 999 won't get you the Emergency Services, and that's not the only thing that's changing!",
	"Nicer ambulances, faster response times and better looking drivers mean they're not just the Emergency Services, they're your Emergency Services.",
	"So, remember the new number! 0118 999! 88199, 9119 725! ... 3!",
	"Hello? I've had a bit of a tumble.",
	"Well that's easy to remember. 0118 999 88199 9119 725! ... 3!",
	"I don't see how they couldn't just keep it as it was. How hard is it to remember 911?",
	"You mean 999. Yes, yes, I mean 999! Yeah, I know. That's the American one, you berk!",
	"I'll put this over here, with the rest of the fire.",
	"0115... no... 0118... no... 0118 999 ... 3. Hello? Is this the emergency services? Then which country am I speaking to? Hello? Hello?",
	"Dear Sir stroke Madam, I am writing to inform you of a fire which has broken out at the premises of...",
	"Dear Sir stroke Madam. Fire, exclamation mark. Fire, exclamation mark. Help me, exclamation mark. 123 Carrendon Road. Looking forward to hearing from you. All the best, Maurice Moss.",
	"I'm a 32 year old IT-man who works in a basement. Yes, I do the whole Lonely Hearts thing!",
	"Shut up, do what I tell you, I'm not interested; these are just some of the things you'll be hearing if you answer this ad. I'm an idiot and I dont care about anyone but myself. P.S. No dogs!",
	"I'm going to murder you... You bloody woman!",
	"Might want to play a bit hard to get.",
	"We don't need no education. Yes you do. You've just used a double negative.",
	"How can you two... Don't Google the question, Moss!",
	"If anyone was ever rude to me, I used to carry their food around in my trousers. Oh my God! Before you brought it to their table? No, after! Of course, before! Why would I do it after?",
	"While he was eating, did you hear anyone laughing? Like... in the kitchen area? Yes! Yes I did, actually, yes I did. That'd be trouser food!",
	"OK. Moss, what did you have for breakfast this morning? Smarties cereal.",
	"Oh my God. I didn't even know Smarties made a cereal. They don't. It's just Smarties in a bowl with milk.",
	"I am a man, he's a man, we're men! Ok, tell me how your feeling. I feel delicate... and annoyed, and... I think I'm ugly.",
	"I've got Aunt Irma visiting. Oh, do you not like Aunt Irma? I've got an aunt like that.",
	"It's my term for my time of the month. Oh. What time of the month? The weekend?",
	"You know, it's high tide. But we're not on the coast. I'm closed for maintenance! Closed for maintenance? I've fallen to the communists! Well, they do have some strong arguments.",
	"Carrie, Moss! First scene in Carrie! Oh. Okay",
	"A gay musical, called Gay. That's quite gay. Gay musical? Aren't all musicals gay? This must be, like, the gayest musical ever.",
	"A story of a young man trying to find his sexuality in the uncaring Thatcher years. Warning: Contains scenes of graphic homoeroticism.",
	"Graphic homoeroticism? Does that mean they're going to get them out?",
	"You're not comfortable with your sexuality? Oh, I'm very comfortable with my sexuality, I just don't want to be slapped in the face with their sexuality.",
	"He's had quite an evening. Someone stole his wheelchair. Did you see who it was? Red bearded man.",
	"How long have you been disabled? Ten years? Ten years, and how did it happen? If that's not a rude question. Acid?",
	"When I started Reynholm Industries, I had just two things in my possession: a dream and 6 million pounds.",
	"Today I have a business empire the like of which the world has never seen the like of which. I hope it doesn't sound arrogant when I say that I am the greatest man in the world!",
	"Unbelievable! Some idiot disabled his firewall, meaning all the computers on Seven are teeming with viruses, plus I've just had to walk all the way down the motherfudging stairs, because the lifts are broken AGAIN!",
}

// index is the base html string for... index
var index = `
<!DOCTYPE html>

        <meta charset="utf-8">
        <title>It Crowd Ipsum</title>

It Crowd Ipsum

Placeholder text taken from The It Crowd

                {{range .Paragraphs}}
{{ . }}
{{ end }}
                <menu>
                <textarea id="text">
{{range .Paragraphs}}{{ . }}

{{ end }}
</textarea>
                    <button type="button" id="copy">Copy?</button>
                    Now press CMD + C / CTRL + C
                </menu>

Inspired by Bluth Ipsum. Made by Kivlor

`

// right up main street
func main() {
	// make sure we have a port
	port := os.Getenv("PORT")

	if port == "" {
		panic("unable to determine port")
	}

	http.HandleFunc("/", root)
	http.ListenAndServe(":"+port, nil)
}

// root is the handler for requests to "/"
func root(w http.ResponseWriter, r *http.Request) {
	// allocate a new html template
	tmpl, err := template.New("home").Parse(index)
	if err != nil {
		panic("unable to parse index")
	}

	// build the template data
	data := struct {
		Paragraphs []string
	}{
		Paragraphs: GenerateLipsum(5),
	}

	// execute the template data
	tmpl.Execute(w, data)
}

// GenerateLipsum will create a number of paragraphs using randome phrases
func GenerateLipsum(count int) []string {
	var lipsum []string
	var paragraph string

	// loop the paragraph count
	for i := 0; i < count; i++ {
		paragraph = ""
		// about 6 phrases makes a goo paragrpah
		for j := 0; j < 6; j++ {
			paragraph += phrases[rand.Intn(len(phrases))]
			paragraph += " "
		}

		// append our paragraph to lipsum
		lipsum = append(lipsum, paragraph)
	}

	// return lipsum
	return lipsum
}
<|endoftext|>


<commit_before>package main

import (
	"bufio"
	"flag"
	"fmt"
	"io"
	"os"

	"github.com/lfkeitel/nitrogen/src/eval"
	"github.com/lfkeitel/nitrogen/src/lexer"
	"github.com/lfkeitel/nitrogen/src/object"
	"github.com/lfkeitel/nitrogen/src/parser"
)

const PROMPT = ">> "

var (
	interactive bool
)

func init() {
	flag.BoolVar(&interactive, "i", false, "Interactive mode")
}

func main() {
	flag.Parse()

	if interactive {
		fmt.Print("Nitrogen Programming Language\n")
		fmt.Print("Type in commands at the prompt\n")
		startRepl(os.Stdin, os.Stdout)
		return
	}

	if len(os.Args) != 2 {
		fmt.Print("No file given")
		os.Exit(1)
	}

	file, err := os.Open(os.Args[1])
	if err != nil {
		fmt.Print(err)
		os.Exit(1)
	}

	l := lexer.New(file)
	p := parser.New(l)
	program := p.ParseProgram()
	if len(p.Errors()) != 0 {
		printParserErrors(os.Stdout, p.Errors())
		os.Exit(1)
	}

	result := eval.Eval(program, object.NewEnvironment())
	if result != nil && result != eval.NULL {
		io.WriteString(os.Stdout, result.Inspect())
	}
}

func startRepl(in io.Reader, out io.Writer) {
	scanner := bufio.NewScanner(in)
	env := object.NewEnvironment()

	for {
		fmt.Fprint(out, PROMPT)
		scanned := scanner.Scan()
		if !scanned {
			return
		}

		line := scanner.Text()
		if line == ".quit" {
			return
		}

		l := lexer.NewString(line)
		p := parser.New(l)

		program := p.ParseProgram()
		if len(p.Errors()) != 0 {
			printParserErrors(out, p.Errors())
			continue
		}

		result := eval.Eval(program, env)
		if result != nil {
			io.WriteString(out, result.Inspect())
			io.WriteString(out, "\n")
		}
	}
}

func printParserErrors(out io.Writer, errors []string) {
	for _, msg := range errors {
		fmt.Fprintf(out, "\t%s\n", msg)
	}
}
<commit_msg>Use the -f flag for executing scripts<commit_after>package main

import (
	"bufio"
	"flag"
	"fmt"
	"io"
	"os"

	"github.com/lfkeitel/nitrogen/src/eval"
	"github.com/lfkeitel/nitrogen/src/lexer"
	"github.com/lfkeitel/nitrogen/src/object"
	"github.com/lfkeitel/nitrogen/src/parser"
)

const PROMPT = ">> "

var (
	interactive bool
	scriptFile  string
)

func init() {
	flag.StringVar(&scriptFile, "f", "", "Filename to execute")
	flag.BoolVar(&interactive, "i", false, "Interactive mode")
}

func main() {
	flag.Parse()

	if interactive {
		fmt.Print("Nitrogen Programming Language\n")
		fmt.Print("Type in commands at the prompt\n")
		startRepl(os.Stdin, os.Stdout)
		return
	}

	if scriptFile == "" {
		fmt.Print("No file given")
		os.Exit(1)
	}

	file, err := os.Open(scriptFile)
	if err != nil {
		fmt.Print(err)
		os.Exit(1)
	}

	l := lexer.New(file)
	p := parser.New(l)
	program := p.ParseProgram()
	if len(p.Errors()) != 0 {
		printParserErrors(os.Stdout, p.Errors())
		os.Exit(1)
	}

	result := eval.Eval(program, object.NewEnvironment())
	if result != nil && result != eval.NULL {
		io.WriteString(os.Stdout, result.Inspect())
	}
}

func startRepl(in io.Reader, out io.Writer) {
	scanner := bufio.NewScanner(in)
	env := object.NewEnvironment()

	for {
		fmt.Fprint(out, PROMPT)
		scanned := scanner.Scan()
		if !scanned {
			return
		}

		line := scanner.Text()
		if line == ".quit" {
			return
		}

		l := lexer.NewString(line)
		p := parser.New(l)

		program := p.ParseProgram()
		if len(p.Errors()) != 0 {
			printParserErrors(out, p.Errors())
			continue
		}

		result := eval.Eval(program, env)
		if result != nil {
			io.WriteString(out, result.Inspect())
			io.WriteString(out, "\n")
		}
	}
}

func printParserErrors(out io.Writer, errors []string) {
	for _, msg := range errors {
		fmt.Fprintf(out, "\t%s\n", msg)
	}
}
<|endoftext|>


<commit_before>package main

import (
	"flag"
	"fmt"
	"log"
	"os"
	"strings"

	"github.com/miku/stardust"
)

func main() {

	distanceFuncMap := map[string]interface{}{
		"hamming":     stardust.HammingDistance,
		"levenshtein": stardust.LevenshteinDistance,
		"ngram":       stardust.NgramSimilarity,
		"jaro":        stardust.JaroSimilarity,
	}

	measure := flag.String("m", "ngram", "distance measure")
	listFuncs := flag.Bool("l", false, "list available measures")

	flag.Parse()

	if *listFuncs {
		for k, _ := range distanceFuncMap {
			fmt.Println(k)
		}
		return
	}

	var PrintUsage = func() {
		fmt.Fprintf(os.Stderr, "Usage: %s [OPTIONS] STRING STRING\n", os.Args[0])
		flag.PrintDefaults()
	}

	if len(flag.Args()) != 2 {
		PrintUsage()
		os.Exit(1)
	}

	// find the right prefix function
	var keys []string
	for k := range distanceFuncMap {
		keys = append(keys, k)
	}
	result := stardust.CompleteString(keys, *measure)
	if len(result) > 1 {
		log.Fatalf("ambiguous name: %s\n", strings.Join(result, ", "))
	} else if len(result) == 0 {
		log.Fatal("no such distance function")
	}
	fn, _ := distanceFuncMap[result[0]]

	a := flag.Args()[0]
	b := flag.Args()[1]

	// we have both int and float functions
	switch fn.(type) {
	default:
		log.Fatal("unknown signature")
	case func(string, string) (float64, error):
		result, err := fn.(func(string, string) (float64, error))(a, b)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Println(result)
	case func(string, string) (int, error):
		result, err := fn.(func(string, string) (int, error))(a, b)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Println(result)
	}
}
<commit_msg>display measures in alphabetical order<commit_after>package main

import (
	"flag"
	"fmt"
	"log"
	"os"
	"sort"
	"strings"

	"github.com/miku/stardust"
)

func main() {

	distanceFuncMap := map[string]interface{}{
		"hamming":     stardust.HammingDistance,
		"levenshtein": stardust.LevenshteinDistance,
		"ngram":       stardust.NgramSimilarity,
		"jaro":        stardust.JaroSimilarity,
	}

	measure := flag.String("m", "ngram", "distance measure")
	listFuncs := flag.Bool("l", false, "list available measures")

	flag.Parse()

	if *listFuncs {
		var keys []string
		for k := range distanceFuncMap {
			keys = append(keys, k)
		}
		sort.Strings(keys)
		for _, k := range keys {
			fmt.Println(k)
		}
		return
	}

	var PrintUsage = func() {
		fmt.Fprintf(os.Stderr, "Usage: %s [OPTIONS] STRING STRING\n", os.Args[0])
		flag.PrintDefaults()
	}

	if len(flag.Args()) != 2 {
		PrintUsage()
		os.Exit(1)
	}

	// find the right prefix function
	var keys []string
	for k := range distanceFuncMap {
		keys = append(keys, k)
	}
	result := stardust.CompleteString(keys, *measure)
	if len(result) > 1 {
		log.Fatalf("ambiguous name: %s\n", strings.Join(result, ", "))
	} else if len(result) == 0 {
		log.Fatal("no such distance function")
	}
	fn, _ := distanceFuncMap[result[0]]

	a := flag.Args()[0]
	b := flag.Args()[1]

	// we have both int and float functions
	switch fn.(type) {
	default:
		log.Fatal("unknown signature")
	case func(string, string) (float64, error):
		result, err := fn.(func(string, string) (float64, error))(a, b)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Println(result)
	case func(string, string) (int, error):
		result, err := fn.(func(string, string) (int, error))(a, b)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Println(result)
	}
}
<|endoftext|>


<commit_before>/*
 * This file is part of the KubeVirt project
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 *
 * Copyright 2021 Red Hat, Inc.
 *
 */

package main

import (
	"os"

	"github.com/spf13/pflag"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

	v1 "kubevirt.io/api/core/v1"
	"kubevirt.io/client-go/log"

	cmdclient "kubevirt.io/kubevirt/pkg/virt-handler/cmd-client"
)

func getGrpcClient() (cmdclient.LauncherClient, error) {
	sockFile := "/run/kubevirt/sockets/launcher-sock"
	client, err := cmdclient.NewClient(sockFile)
	if err != nil {
		log.Log.Reason(err).Error("Failed to connect launcher")
		os.Exit(1)
	}

	return client, err
}

func main() {
	log.InitializeLogging("freezer")
	log.Log.Info("Starting...")

	freeze := pflag.Bool("freeze", false, "Freeze VM")
	unfreeze := pflag.Bool("unfreeze", false, "Freeze VM")
	name := pflag.String("name", "", "Name of the VirtualMachineInstance")
	namespace := pflag.String("namespace", "", "Namespace of the VirtualMachineInstance")
	unfreezeTimeoutSeconds := pflag.Int32("unfreezeTimeoutSeconds", 300, "Timeout in seconds to automatically unfreeze the VirtualMachineInstance")

	pflag.Parse()

	if !*freeze && !*unfreeze {
		log.Log.Errorf("Use either --freeze or --unfreeze")
		os.Exit(1)
	}
	if name == nil || namespace == nil {
		log.Log.Errorf("Both name and namespace flags must be provided")
		os.Exit(1)
	}

	vmi := &v1.VirtualMachineInstance{
		ObjectMeta: metav1.ObjectMeta{
			Name:      *name,
			Namespace: *namespace,
		},
	}

	client, err := getGrpcClient()
	if err != nil {
		log.Log.Reason(err).Error("Failed to connect launcher")
		os.Exit(1)
	}

	info, err := client.GetGuestInfo()
	if err != nil {
		log.Log.Reason(err).Error("Failed to get guest info")
		os.Exit(1)
	}

	log.Log.Infof("Guest agent version is %s", info.GAVersion)
	if info.GAVersion == "" {
		log.Log.Info("No guest agent, exiting")
		os.Exit(0)
	}

	if *freeze {
		err = client.FreezeVirtualMachine(vmi, *unfreezeTimeoutSeconds)
		if err != nil {
			log.Log.Reason(err).Error("Freezeing VMI failed")
			os.Exit(1)
		}
	} else {
		err = client.UnfreezeVirtualMachine(vmi)
		if err != nil {
			log.Log.Reason(err).Error("Unfreezeing VMI failed")
			os.Exit(1)
		}
	}

	log.Log.Info("Exiting...")
}
<commit_msg>move virt-freezer log message<commit_after>/*
 * This file is part of the KubeVirt project
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 *
 * Copyright 2021 Red Hat, Inc.
 *
 */

package main

import (
	"os"

	"github.com/spf13/pflag"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

	v1 "kubevirt.io/api/core/v1"
	"kubevirt.io/client-go/log"

	cmdclient "kubevirt.io/kubevirt/pkg/virt-handler/cmd-client"
)

func getGrpcClient() (cmdclient.LauncherClient, error) {
	sockFile := "/run/kubevirt/sockets/launcher-sock"
	client, err := cmdclient.NewClient(sockFile)
	if err != nil {
		log.Log.Reason(err).Error("Failed to connect launcher")
		os.Exit(1)
	}

	return client, err
}

func main() {
	log.InitializeLogging("freezer")
	log.Log.Info("Starting...")

	freeze := pflag.Bool("freeze", false, "Freeze VM")
	unfreeze := pflag.Bool("unfreeze", false, "Freeze VM")
	name := pflag.String("name", "", "Name of the VirtualMachineInstance")
	namespace := pflag.String("namespace", "", "Namespace of the VirtualMachineInstance")
	unfreezeTimeoutSeconds := pflag.Int32("unfreezeTimeoutSeconds", 300, "Timeout in seconds to automatically unfreeze the VirtualMachineInstance")

	pflag.Parse()

	if !*freeze && !*unfreeze {
		log.Log.Errorf("Use either --freeze or --unfreeze")
		os.Exit(1)
	}
	if name == nil || namespace == nil {
		log.Log.Errorf("Both name and namespace flags must be provided")
		os.Exit(1)
	}

	vmi := &v1.VirtualMachineInstance{
		ObjectMeta: metav1.ObjectMeta{
			Name:      *name,
			Namespace: *namespace,
		},
	}

	client, err := getGrpcClient()
	if err != nil {
		log.Log.Reason(err).Error("Failed to connect launcher")
		os.Exit(1)
	}

	info, err := client.GetGuestInfo()
	if err != nil {
		log.Log.Reason(err).Error("Failed to get guest info")
		os.Exit(1)
	}

	if info.GAVersion == "" {
		log.Log.Info("No guest agent, exiting")
		os.Exit(0)
	}

	log.Log.Infof("Guest agent version is %s", info.GAVersion)

	if *freeze {
		err = client.FreezeVirtualMachine(vmi, *unfreezeTimeoutSeconds)
		if err != nil {
			log.Log.Reason(err).Error("Freezeing VMI failed")
			os.Exit(1)
		}
	} else {
		err = client.UnfreezeVirtualMachine(vmi)
		if err != nil {
			log.Log.Reason(err).Error("Unfreezeing VMI failed")
			os.Exit(1)
		}
	}

	log.Log.Info("Exiting...")
}
<|endoftext|>


<commit_before>package main

import (
	"flag"
	"fmt"
	"log"
	"os"
	"os/exec"
	"strings"
	"time"
)

const usage = `
  Usage: piccolo <interval> <command>
`

// const green1 = "\033[0;40;32m"

const (
	normal        = "\033[0m"
	blod          = "\033[1m"
	italics       = "\033[3m" // test failed
	underline     = "\033[4m"
	inverse       = "\033[7m"
	strikethrough = "\033[9m" // test failed

	foreBlack   = "\033[30m"
	foreRed     = "\033[31m"
	foreGreen   = "\033[32m"
	foreYellow  = "\033[33m"
	foreBlue    = "\033[34m"
	forePurple  = "\033[35m"
	foreCyan    = "\033[36m"
	foreWhite   = "\033[37m"
	foreDefault = "\033[39m"

	backBlack   = "\033[40m"
	backRed     = "\033[41m"
	backGreen   = "\033[42m"
	backYellow  = "\033[43m"
	backBlue    = "\033[44m"
	backPurple  = "\033[45m"
	backCyan    = "\033[46m"
	backWhite   = "\033[47m"
	backDefault = "\033[49m"
)

var (
	flags = flag.NewFlagSet("every", flag.ContinueOnError)
	exit  = flags.Bool("exit", false, "")
)

func printUsage() {
	fmt.Println(usage)
	os.Exit(0)
}

func checkErr(err error) {
	if err != nil {
		log.Fatalf("%sError: %s", foreRed, err)
	}
}

func main() {
	flags.Usage = printUsage
	flags.Parse(os.Args[1:])
	argv := flags.Args()

	if len(argv) < 1 {
		checkErr(fmt.Errorf("<interval> required"))
	}

	interval, err := time.ParseDuration(argv[0])
	checkErr(err)

	if len(argv) < 2 {
		checkErr(fmt.Errorf("<command> required"))
	}

	cmd := strings.Join(argv[1:], " ")

	log.Printf("%severy %s running %s", forePurple, interval, cmd)

	for {
		time.Sleep(interval)
		start := time.Now()
		log.Printf("%sexec `%s`", foreGreen, cmd)
		proc := exec.Command("/bin/sh", "-c", cmd)

		proc.Start()
		err := proc.Wait()
		ps := proc.ProcessState

		if err != nil {
			log.Printf("pid %d failed with %s", ps.Pid(), ps.String())
			if *exit {
				os.Exit(1)
			}
			continue
		}
		log.Printf("pid %d completed in %s", ps.Pid(), time.Since(start))
	}
}
<commit_msg>update output format<commit_after>package main

import (
	"flag"
	"fmt"
	"log"
	"os"
	"os/exec"
	"strings"
	"time"
)

const usage = `
  Usage: piccolo <interval> <command>
`

// const green1 = "\033[0;40;32m"

const (
	normal        = "\033[0m"
	blod          = "\033[1m"
	italics       = "\033[3m" // test failed
	underline     = "\033[4m"
	inverse       = "\033[7m"
	strikethrough = "\033[9m" // test failed

	foreBlack   = "\033[30m"
	foreRed     = "\033[31m"
	foreGreen   = "\033[32m"
	foreYellow  = "\033[33m"
	foreBlue    = "\033[34m"
	forePurple  = "\033[35m"
	foreCyan    = "\033[36m"
	foreWhite   = "\033[37m"
	foreDefault = "\033[39m"

	backBlack   = "\033[40m"
	backRed     = "\033[41m"
	backGreen   = "\033[42m"
	backYellow  = "\033[43m"
	backBlue    = "\033[44m"
	backPurple  = "\033[45m"
	backCyan    = "\033[46m"
	backWhite   = "\033[47m"
	backDefault = "\033[49m"
)

var (
	flags = flag.NewFlagSet("every", flag.ContinueOnError)
	exit  = flags.Bool("exit", false, "")
)

func printUsage() {
	fmt.Println(usage)
	os.Exit(0)
}

func checkErr(err error) {
	if err != nil {
		log.Fatalf("%sError: %s", foreRed, err)
	}
}

func main() {
	flags.Usage = printUsage
	flags.Parse(os.Args[1:])
	argv := flags.Args()

	if len(argv) < 1 {
		checkErr(fmt.Errorf("<interval> required"))
	}

	interval, err := time.ParseDuration(argv[0])
	checkErr(err)

	if len(argv) < 2 {
		checkErr(fmt.Errorf("<command> required"))
	}

	cmd := strings.Join(argv[1:], " ")

	log.Printf("%severy %s running %s%s", forePurple, interval, cmd, normal)

	for {
		time.Sleep(interval)
		start := time.Now()
		log.Printf("%sexec `%s`%s", foreGreen, cmd, normal)
		proc := exec.Command("/bin/sh", "-c", cmd)

		proc.Start()
		err := proc.Wait()
		ps := proc.ProcessState

		if err != nil {
			log.Printf("pid %d failed with %s", ps.Pid(), ps.String())
			if *exit {
				os.Exit(1)
			}
			continue
		}
		log.Printf("pid %d completed in %s", ps.Pid(), time.Since(start))
	}
}
<|endoftext|>


<commit_before>package dht

import (
	"sync"

	context "github.com/jbenet/go-ipfs/Godeps/_workspace/src/code.google.com/p/go.net/context"

	inet "github.com/jbenet/go-ipfs/net"
	peer "github.com/jbenet/go-ipfs/peer"
	"github.com/jbenet/go-ipfs/routing"
	pb "github.com/jbenet/go-ipfs/routing/dht/pb"
	kb "github.com/jbenet/go-ipfs/routing/kbucket"
	u "github.com/jbenet/go-ipfs/util"
)

// asyncQueryBuffer is the size of buffered channels in async queries. This
// buffer allows multiple queries to execute simultaneously, return their
// results and continue querying closer peers. Note that different query
// results will wait for the channel to drain.
var asyncQueryBuffer = 10

// This file implements the Routing interface for the IpfsDHT struct.

// Basic Put/Get

// PutValue adds value corresponding to given Key.
// This is the top level "Store" operation of the DHT
func (dht *IpfsDHT) PutValue(ctx context.Context, key u.Key, value []byte) error {
	log.Debugf("PutValue %s", key)
	err := dht.putLocal(key, value)
	if err != nil {
		return err
	}

	rec, err := dht.makePutRecord(key, value)
	if err != nil {
		log.Error("Creation of record failed!")
		return err
	}

	var peers []peer.Peer
	for _, route := range dht.routingTables {
		npeers := route.NearestPeers(kb.ConvertKey(key), KValue)
		peers = append(peers, npeers...)
	}

	query := newQuery(key, dht.dialer, func(ctx context.Context, p peer.Peer) (*dhtQueryResult, error) {
		log.Debugf("%s PutValue qry part %v", dht.self, p)
		err := dht.putValueToNetwork(ctx, p, string(key), rec)
		if err != nil {
			return nil, err
		}
		return &dhtQueryResult{success: true}, nil
	})

	_, err = query.Run(ctx, peers)
	return err
}

// GetValue searches for the value corresponding to given Key.
// If the search does not succeed, a multiaddr string of a closer peer is
// returned along with util.ErrSearchIncomplete
func (dht *IpfsDHT) GetValue(ctx context.Context, key u.Key) ([]byte, error) {
	log.Debugf("Get Value [%s]", key)

	// If we have it local, dont bother doing an RPC!
	// NOTE: this might not be what we want to do...
	val, err := dht.getLocal(key)
	if err == nil {
		log.Debug("Got value locally!")
		return val, nil
	}

	// get closest peers in the routing tables
	routeLevel := 0
	closest := dht.routingTables[routeLevel].NearestPeers(kb.ConvertKey(key), PoolSize)
	if closest == nil || len(closest) == 0 {
		log.Warning("Got no peers back from routing table!")
		return nil, kb.ErrLookupFailure
	}

	// setup the Query
	query := newQuery(key, dht.dialer, func(ctx context.Context, p peer.Peer) (*dhtQueryResult, error) {

		val, peers, err := dht.getValueOrPeers(ctx, p, key, routeLevel)
		if err != nil {
			return nil, err
		}

		res := &dhtQueryResult{value: val, closerPeers: peers}
		if val != nil {
			res.success = true
		}

		return res, nil
	})

	// run it!
	result, err := query.Run(ctx, closest)
	if err != nil {
		return nil, err
	}

	log.Debugf("GetValue %v %v", key, result.value)
	if result.value == nil {
		return nil, routing.ErrNotFound
	}

	return result.value, nil
}

// Value provider layer of indirection.
// This is what DSHTs (Coral and MainlineDHT) do to store large values in a DHT.

// Provide makes this node announce that it can provide a value for the given key
func (dht *IpfsDHT) Provide(ctx context.Context, key u.Key) error {

	dht.providers.AddProvider(key, dht.self)
	peers := dht.routingTables[0].NearestPeers(kb.ConvertKey(key), PoolSize)
	if len(peers) == 0 {
		return nil
	}

	//TODO FIX: this doesn't work! it needs to be sent to the actual nearest peers.
	// `peers` are the closest peers we have, not the ones that should get the value.
	for _, p := range peers {
		err := dht.putProvider(ctx, p, string(key))
		if err != nil {
			return err
		}
	}
	return nil
}

// FindProvidersAsync is the same thing as FindProviders, but returns a channel.
// Peers will be returned on the channel as soon as they are found, even before
// the search query completes.
func (dht *IpfsDHT) FindProvidersAsync(ctx context.Context, key u.Key, count int) <-chan peer.Peer {
	log.Event(ctx, "findProviders", &key)
	peerOut := make(chan peer.Peer, count)
	go dht.findProvidersAsyncRoutine(ctx, key, count, peerOut)
	return peerOut
}

func (dht *IpfsDHT) findProvidersAsyncRoutine(ctx context.Context, key u.Key, count int, peerOut chan peer.Peer) {
	defer close(peerOut)

	ps := newPeerSet()
	provs := dht.providers.GetProviders(ctx, key)
	for _, p := range provs {
		count--
		// NOTE: assuming that this list of peers is unique
		ps.Add(p)
		select {
		case peerOut <- p:
		case <-ctx.Done():
			return
		}
		if count <= 0 {
			return
		}
	}

	// setup the Query
	query := newQuery(key, dht.dialer, func(ctx context.Context, p peer.Peer) (*dhtQueryResult, error) {

		pmes, err := dht.findProvidersSingle(ctx, p, key, 0)
		if err != nil {
			return nil, err
		}

		provs, errs := pb.PBPeersToPeers(dht.peerstore, pmes.GetProviderPeers())
		for _, err := range errs {
			if err != nil {
				log.Warning(err)
			}
		}

		// Add unique providers from request, up to 'count'
		for _, prov := range provs {
			if ps.Contains(prov) {
				continue
			}
			select {
			case peerOut <- prov:
			case <-ctx.Done():
				log.Error("Context timed out sending more providers")
				return nil, ctx.Err()
			}
			ps.Add(prov)
			if ps.Size() >= count {
				return &dhtQueryResult{success: true}, nil
			}
		}

		// Give closer peers back to the query to be queried
		closer := pmes.GetCloserPeers()
		clpeers, errs := pb.PBPeersToPeers(dht.peerstore, closer)
		for _, err := range errs {
			if err != nil {
				log.Warning(err)
			}
		}

		return &dhtQueryResult{closerPeers: clpeers}, nil
	})

	peers := dht.routingTables[0].NearestPeers(kb.ConvertKey(key), AlphaValue)
	_, err := query.Run(ctx, peers)
	if err != nil {
		log.Errorf("FindProviders Query error: %s", err)
	}
}

func (dht *IpfsDHT) addPeerListAsync(ctx context.Context, k u.Key, peers []*pb.Message_Peer, ps *peerSet, count int, out chan peer.Peer) {
	var wg sync.WaitGroup
	for _, pbp := range peers {
		wg.Add(1)
		go func(mp *pb.Message_Peer) {
			defer wg.Done()
			// construct new peer
			p, err := dht.ensureConnectedToPeer(ctx, mp)
			if err != nil {
				log.Errorf("%s", err)
				return
			}
			if p == nil {
				log.Error("Got nil peer from ensureConnectedToPeer")
				return
			}

			dht.providers.AddProvider(k, p)
			if ps.AddIfSmallerThan(p, count) {
				select {
				case out <- p:
				case <-ctx.Done():
					return
				}
			} else if ps.Size() >= count {
				return
			}
		}(pbp)
	}
	wg.Wait()
}

// FindPeer searches for a peer with given ID.
func (dht *IpfsDHT) FindPeer(ctx context.Context, id peer.ID) (peer.Peer, error) {

	// Check if were already connected to them
	p, _ := dht.FindLocal(id)
	if p != nil {
		return p, nil
	}

	routeLevel := 0
	closest := dht.routingTables[routeLevel].NearestPeers(kb.ConvertPeerID(id), AlphaValue)
	if closest == nil || len(closest) == 0 {
		return nil, kb.ErrLookupFailure
	}

	// Sanity...
	for _, p := range closest {
		if p.ID().Equal(id) {
			log.Error("Found target peer in list of closest peers...")
			return p, nil
		}
	}

	// setup the Query
	query := newQuery(u.Key(id), dht.dialer, func(ctx context.Context, p peer.Peer) (*dhtQueryResult, error) {

		pmes, err := dht.findPeerSingle(ctx, p, id, routeLevel)
		if err != nil {
			return nil, err
		}

		closer := pmes.GetCloserPeers()
		clpeers, errs := pb.PBPeersToPeers(dht.peerstore, closer)
		for _, err := range errs {
			if err != nil {
				log.Warning(err)
			}
		}

		// see it we got the peer here
		for _, np := range clpeers {
			if string(np.ID()) == string(id) {
				return &dhtQueryResult{
					peer:    np,
					success: true,
				}, nil
			}
		}

		return &dhtQueryResult{closerPeers: clpeers}, nil
	})

	// run it!
	result, err := query.Run(ctx, closest)
	if err != nil {
		return nil, err
	}

	log.Debugf("FindPeer %v %v", id, result.success)
	if result.peer == nil {
		return nil, routing.ErrNotFound
	}

	return result.peer, nil
}

// FindPeersConnectedToPeer searches for peers directly connected to a given peer.
func (dht *IpfsDHT) FindPeersConnectedToPeer(ctx context.Context, id peer.ID) (<-chan peer.Peer, error) {

	peerchan := make(chan peer.Peer, asyncQueryBuffer)
	peersSeen := map[string]peer.Peer{}

	routeLevel := 0
	closest := dht.routingTables[routeLevel].NearestPeers(kb.ConvertPeerID(id), AlphaValue)
	if closest == nil || len(closest) == 0 {
		return nil, kb.ErrLookupFailure
	}

	// setup the Query
	query := newQuery(u.Key(id), dht.dialer, func(ctx context.Context, p peer.Peer) (*dhtQueryResult, error) {

		pmes, err := dht.findPeerSingle(ctx, p, id, routeLevel)
		if err != nil {
			return nil, err
		}

		var clpeers []peer.Peer
		closer := pmes.GetCloserPeers()
		for _, pbp := range closer {
			// skip peers already seen
			if _, found := peersSeen[string(pbp.GetId())]; found {
				continue
			}

			// skip peers that fail to unmarshal
			p, err := pb.PBPeerToPeer(dht.peerstore, pbp)
			if err != nil {
				log.Warning(err)
				continue
			}

			// if peer is connected, send it to our client.
			if pb.Connectedness(*pbp.Connection) == inet.Connected {
				select {
				case <-ctx.Done():
					return nil, ctx.Err()
				case peerchan <- p:
				}
			}

			peersSeen[string(p.ID())] = p

			// if peer is the peer we're looking for, don't bother querying it.
			if pb.Connectedness(*pbp.Connection) != inet.Connected {
				clpeers = append(clpeers, p)
			}
		}

		return &dhtQueryResult{closerPeers: clpeers}, nil
	})

	// run it! run it asynchronously to gen peers as results are found.
	// this does no error checking
	go func() {
		if _, err := query.Run(ctx, closest); err != nil {
			log.Error(err)
		}

		// close the peerchan channel when done.
		close(peerchan)
	}()

	return peerchan, nil
}

// Ping a peer, log the time it took
func (dht *IpfsDHT) Ping(ctx context.Context, p peer.Peer) error {
	// Thoughts: maybe this should accept an ID and do a peer lookup?
	log.Debugf("ping %s start", p)

	pmes := pb.NewMessage(pb.Message_PING, "", 0)
	_, err := dht.sendRequest(ctx, p, pmes)
	log.Debugf("ping %s end (err = %s)", p, err)
	return err
}
<commit_msg>changes from PR<commit_after>package dht

import (
	"sync"

	context "github.com/jbenet/go-ipfs/Godeps/_workspace/src/code.google.com/p/go.net/context"

	inet "github.com/jbenet/go-ipfs/net"
	peer "github.com/jbenet/go-ipfs/peer"
	"github.com/jbenet/go-ipfs/routing"
	pb "github.com/jbenet/go-ipfs/routing/dht/pb"
	kb "github.com/jbenet/go-ipfs/routing/kbucket"
	u "github.com/jbenet/go-ipfs/util"
)

// asyncQueryBuffer is the size of buffered channels in async queries. This
// buffer allows multiple queries to execute simultaneously, return their
// results and continue querying closer peers. Note that different query
// results will wait for the channel to drain.
var asyncQueryBuffer = 10

// This file implements the Routing interface for the IpfsDHT struct.

// Basic Put/Get

// PutValue adds value corresponding to given Key.
// This is the top level "Store" operation of the DHT
func (dht *IpfsDHT) PutValue(ctx context.Context, key u.Key, value []byte) error {
	log.Debugf("PutValue %s", key)
	err := dht.putLocal(key, value)
	if err != nil {
		return err
	}

	rec, err := dht.makePutRecord(key, value)
	if err != nil {
		log.Error("Creation of record failed!")
		return err
	}

	var peers []peer.Peer
	for _, route := range dht.routingTables {
		npeers := route.NearestPeers(kb.ConvertKey(key), KValue)
		peers = append(peers, npeers...)
	}

	query := newQuery(key, dht.dialer, func(ctx context.Context, p peer.Peer) (*dhtQueryResult, error) {
		log.Debugf("%s PutValue qry part %v", dht.self, p)
		err := dht.putValueToNetwork(ctx, p, string(key), rec)
		if err != nil {
			return nil, err
		}
		return &dhtQueryResult{success: true}, nil
	})

	_, err = query.Run(ctx, peers)
	return err
}

// GetValue searches for the value corresponding to given Key.
// If the search does not succeed, a multiaddr string of a closer peer is
// returned along with util.ErrSearchIncomplete
func (dht *IpfsDHT) GetValue(ctx context.Context, key u.Key) ([]byte, error) {
	log.Debugf("Get Value [%s]", key)

	// If we have it local, dont bother doing an RPC!
	// NOTE: this might not be what we want to do...
	val, err := dht.getLocal(key)
	if err == nil {
		log.Debug("Got value locally!")
		return val, nil
	}

	// get closest peers in the routing tables
	routeLevel := 0
	closest := dht.routingTables[routeLevel].NearestPeers(kb.ConvertKey(key), PoolSize)
	if closest == nil || len(closest) == 0 {
		log.Warning("Got no peers back from routing table!")
		return nil, kb.ErrLookupFailure
	}

	// setup the Query
	query := newQuery(key, dht.dialer, func(ctx context.Context, p peer.Peer) (*dhtQueryResult, error) {

		val, peers, err := dht.getValueOrPeers(ctx, p, key, routeLevel)
		if err != nil {
			return nil, err
		}

		res := &dhtQueryResult{value: val, closerPeers: peers}
		if val != nil {
			res.success = true
		}

		return res, nil
	})

	// run it!
	result, err := query.Run(ctx, closest)
	if err != nil {
		return nil, err
	}

	log.Debugf("GetValue %v %v", key, result.value)
	if result.value == nil {
		return nil, routing.ErrNotFound
	}

	return result.value, nil
}

// Value provider layer of indirection.
// This is what DSHTs (Coral and MainlineDHT) do to store large values in a DHT.

// Provide makes this node announce that it can provide a value for the given key
func (dht *IpfsDHT) Provide(ctx context.Context, key u.Key) error {

	dht.providers.AddProvider(key, dht.self)
	peers := dht.routingTables[0].NearestPeers(kb.ConvertKey(key), PoolSize)
	if len(peers) == 0 {
		return nil
	}

	//TODO FIX: this doesn't work! it needs to be sent to the actual nearest peers.
	// `peers` are the closest peers we have, not the ones that should get the value.
	for _, p := range peers {
		err := dht.putProvider(ctx, p, string(key))
		if err != nil {
			return err
		}
	}
	return nil
}

// FindProvidersAsync is the same thing as FindProviders, but returns a channel.
// Peers will be returned on the channel as soon as they are found, even before
// the search query completes.
func (dht *IpfsDHT) FindProvidersAsync(ctx context.Context, key u.Key, count int) <-chan peer.Peer {
	log.Event(ctx, "findProviders", &key)
	peerOut := make(chan peer.Peer, count)
	go dht.findProvidersAsyncRoutine(ctx, key, count, peerOut)
	return peerOut
}

func (dht *IpfsDHT) findProvidersAsyncRoutine(ctx context.Context, key u.Key, count int, peerOut chan peer.Peer) {
	defer close(peerOut)

	ps := newPeerSet()
	provs := dht.providers.GetProviders(ctx, key)
	for _, p := range provs {
		// NOTE: assuming that this list of peers is unique
		if ps.AddIfSmallerThan(p, count) {
			select {
			case peerOut <- p:
			case <-ctx.Done():
				return
			}
		}

		// If we have enough peers locally, dont bother with remote RPC
		if ps.Size() >= count {
			return
		}
	}

	// setup the Query
	query := newQuery(key, dht.dialer, func(ctx context.Context, p peer.Peer) (*dhtQueryResult, error) {

		pmes, err := dht.findProvidersSingle(ctx, p, key, 0)
		if err != nil {
			return nil, err
		}

		provs, errs := pb.PBPeersToPeers(dht.peerstore, pmes.GetProviderPeers())
		for _, err := range errs {
			if err != nil {
				log.Warning(err)
			}
		}

		// Add unique providers from request, up to 'count'
		for _, prov := range provs {
			if ps.AddIfSmallerThan(prov, count) {
				select {
				case peerOut <- prov:
				case <-ctx.Done():
					log.Error("Context timed out sending more providers")
					return nil, ctx.Err()
				}
			}
			if ps.Size() >= count {
				return &dhtQueryResult{success: true}, nil
			}
		}

		// Give closer peers back to the query to be queried
		closer := pmes.GetCloserPeers()
		clpeers, errs := pb.PBPeersToPeers(dht.peerstore, closer)
		for _, err := range errs {
			if err != nil {
				log.Warning(err)
			}
		}

		return &dhtQueryResult{closerPeers: clpeers}, nil
	})

	peers := dht.routingTables[0].NearestPeers(kb.ConvertKey(key), AlphaValue)
	_, err := query.Run(ctx, peers)
	if err != nil {
		log.Errorf("FindProviders Query error: %s", err)
	}
}

func (dht *IpfsDHT) addPeerListAsync(ctx context.Context, k u.Key, peers []*pb.Message_Peer, ps *peerSet, count int, out chan peer.Peer) {
	var wg sync.WaitGroup
	for _, pbp := range peers {
		wg.Add(1)
		go func(mp *pb.Message_Peer) {
			defer wg.Done()
			// construct new peer
			p, err := dht.ensureConnectedToPeer(ctx, mp)
			if err != nil {
				log.Errorf("%s", err)
				return
			}
			if p == nil {
				log.Error("Got nil peer from ensureConnectedToPeer")
				return
			}

			dht.providers.AddProvider(k, p)
			if ps.AddIfSmallerThan(p, count) {
				select {
				case out <- p:
				case <-ctx.Done():
					return
				}
			} else if ps.Size() >= count {
				return
			}
		}(pbp)
	}
	wg.Wait()
}

// FindPeer searches for a peer with given ID.
func (dht *IpfsDHT) FindPeer(ctx context.Context, id peer.ID) (peer.Peer, error) {

	// Check if were already connected to them
	p, _ := dht.FindLocal(id)
	if p != nil {
		return p, nil
	}

	routeLevel := 0
	closest := dht.routingTables[routeLevel].NearestPeers(kb.ConvertPeerID(id), AlphaValue)
	if closest == nil || len(closest) == 0 {
		return nil, kb.ErrLookupFailure
	}

	// Sanity...
	for _, p := range closest {
		if p.ID().Equal(id) {
			log.Error("Found target peer in list of closest peers...")
			return p, nil
		}
	}

	// setup the Query
	query := newQuery(u.Key(id), dht.dialer, func(ctx context.Context, p peer.Peer) (*dhtQueryResult, error) {

		pmes, err := dht.findPeerSingle(ctx, p, id, routeLevel)
		if err != nil {
			return nil, err
		}

		closer := pmes.GetCloserPeers()
		clpeers, errs := pb.PBPeersToPeers(dht.peerstore, closer)
		for _, err := range errs {
			if err != nil {
				log.Warning(err)
			}
		}

		// see it we got the peer here
		for _, np := range clpeers {
			if string(np.ID()) == string(id) {
				return &dhtQueryResult{
					peer:    np,
					success: true,
				}, nil
			}
		}

		return &dhtQueryResult{closerPeers: clpeers}, nil
	})

	// run it!
	result, err := query.Run(ctx, closest)
	if err != nil {
		return nil, err
	}

	log.Debugf("FindPeer %v %v", id, result.success)
	if result.peer == nil {
		return nil, routing.ErrNotFound
	}

	return result.peer, nil
}

// FindPeersConnectedToPeer searches for peers directly connected to a given peer.
func (dht *IpfsDHT) FindPeersConnectedToPeer(ctx context.Context, id peer.ID) (<-chan peer.Peer, error) {

	peerchan := make(chan peer.Peer, asyncQueryBuffer)
	peersSeen := map[string]peer.Peer{}

	routeLevel := 0
	closest := dht.routingTables[routeLevel].NearestPeers(kb.ConvertPeerID(id), AlphaValue)
	if closest == nil || len(closest) == 0 {
		return nil, kb.ErrLookupFailure
	}

	// setup the Query
	query := newQuery(u.Key(id), dht.dialer, func(ctx context.Context, p peer.Peer) (*dhtQueryResult, error) {

		pmes, err := dht.findPeerSingle(ctx, p, id, routeLevel)
		if err != nil {
			return nil, err
		}

		var clpeers []peer.Peer
		closer := pmes.GetCloserPeers()
		for _, pbp := range closer {
			// skip peers already seen
			if _, found := peersSeen[string(pbp.GetId())]; found {
				continue
			}

			// skip peers that fail to unmarshal
			p, err := pb.PBPeerToPeer(dht.peerstore, pbp)
			if err != nil {
				log.Warning(err)
				continue
			}

			// if peer is connected, send it to our client.
			if pb.Connectedness(*pbp.Connection) == inet.Connected {
				select {
				case <-ctx.Done():
					return nil, ctx.Err()
				case peerchan <- p:
				}
			}

			peersSeen[string(p.ID())] = p

			// if peer is the peer we're looking for, don't bother querying it.
			if pb.Connectedness(*pbp.Connection) != inet.Connected {
				clpeers = append(clpeers, p)
			}
		}

		return &dhtQueryResult{closerPeers: clpeers}, nil
	})

	// run it! run it asynchronously to gen peers as results are found.
	// this does no error checking
	go func() {
		if _, err := query.Run(ctx, closest); err != nil {
			log.Error(err)
		}

		// close the peerchan channel when done.
		close(peerchan)
	}()

	return peerchan, nil
}

// Ping a peer, log the time it took
func (dht *IpfsDHT) Ping(ctx context.Context, p peer.Peer) error {
	// Thoughts: maybe this should accept an ID and do a peer lookup?
	log.Debugf("ping %s start", p)

	pmes := pb.NewMessage(pb.Message_PING, "", 0)
	_, err := dht.sendRequest(ctx, p, pmes)
	log.Debugf("ping %s end (err = %s)", p, err)
	return err
}
<|endoftext|>


<commit_before>package picolog

import (
	"bufio"
	"fmt"
	"log"
	"log/syslog"
	"os"
	"strings"
)

// Logger is a leveled logger type.
type Logger struct {
	logLevel    syslog.Priority
	logger      *log.Logger
	writer      *bufio.Writer
	initialized bool
}

// ParseLogLevel takes a string and returns a syslog.Priority according
// to the standard syslog string representation. Not case-sensitive.
func ParseLogLevel(level string) (syslog.Priority, error) {
	level = strings.ToLower(level)
	switch {
	case level == "emerg":
		return syslog.LOG_EMERG, nil
	case level == "alert":
		return syslog.LOG_ALERT, nil
	case level == "crit":
		return syslog.LOG_CRIT, nil
	case level == "err":
		return syslog.LOG_ERR, nil
	case level == "warning":
		return syslog.LOG_WARNING, nil
	case level == "notice":
		return syslog.LOG_NOTICE, nil
	case level == "info":
		return syslog.LOG_INFO, nil
	case level == "debug":
		return syslog.LOG_DEBUG, nil
	}
	return syslog.Priority(0), fmt.Errorf("Invalid log level: %s", level)
}

// Return a new Logger. logLevel is a syslog log level,
// subpackage is used to construct the log prefix, and dest is where to
// write the log to.
func NewLogger(logLevel syslog.Priority, subpackage string, dest *os.File) *Logger {
	logger := new(Logger)
	logger.logLevel = logLevel
	flags := log.Ldate | log.Ltime
	// If logging at DEBUG, include file paths and line numbers
	if logLevel == syslog.LOG_DEBUG {
		flags |= log.Lshortfile
	}
	prefix := fmt.Sprintf("[%s] ", subpackage)
	logger.writer = bufio.NewWriter(dest)
	logger.logger = log.New(logger.writer, prefix, flags)
	logger.initialized = true
	return logger
}

// NewDefaultLogger returns a picolog.Logger initialized with workable
// defaults (outputs to stderr, prefix "default", priority DEBUG).
// Useful as a fallback when a logger hasn't been initialized.
func NewDefaultLogger() *Logger {
	return NewLogger(syslog.LOG_DEBUG, "default", os.Stderr)
}

// initializeDefaultLogger takes a (possibly nil) *Logger and allocates
// and assigns a default logger as returned by NewDefaultLogger.
func (l *Logger) initializeDefaultLogger() {
	defaultLogger := NewDefaultLogger()
	l = defaultLogger
}

// ensureInitialized checks if the initialized flag has been set for l,
// and if not initializes a default logger.
func (l *Logger) ensureInitialized() {
	if !l.initialized {
		l.initializeDefaultLogger()
	}
}

// Printf is the lowest-level output function of our Logger. Will use a
// default logger if l is not initialized.
func (l *Logger) Printf(format string, level syslog.Priority, v ...interface{}) {
	l.ensureInitialized()
	if level <= l.logLevel {
		msg := fmt.Sprintf(format, v...)
		// We use logger.Output rather than logger.Printf
		// so we can pass a custom calldepth for file
		// {path,line}-resolution purposes (the default of 2
		// is only useful when using the Logger type directly).
		l.logger.Output(3, msg)
		l.writer.Flush()
	}
}

// Debugf logs one printf-formatted message at LOG_DEBUG.
func (l *Logger) Debugf(format string, v ...interface{}) {
	l.Printf(format, syslog.LOG_DEBUG, v...)
}

// Errorf logs one printf-formatted message at LOG_ERR.
func (l *Logger) Errorf(format string, v ...interface{}) {
	l.Printf(format, syslog.LOG_ERR, v...)
}

// Warningf logs one printf-formatted message at LOG_WARNING.
func (l *Logger) Warningf(format string, v ...interface{}) {
	l.Printf(format, syslog.LOG_WARNING, v...)
}

// Fatalf logs one printf-formatted message at LOG_CRIT, and then exits
// with an error code.
func (l *Logger) Fatalf(format string, v ...interface{}) {
	l.Printf(format, syslog.LOG_CRIT, v...)
	os.Exit(1)
}

// Infof logs one printf-formatted message at LOG_INFO.
func (l *Logger) Infof(format string, v ...interface{}) {
	l.Printf(format, syslog.LOG_INFO, v...)
}
<commit_msg>Support for subloggers<commit_after>package picolog

import (
	"bufio"
	"fmt"
	"log"
	"log/syslog"
	"os"
	"strings"
)

// Logger is a leveled logger type.
type Logger struct {
	logLevel    syslog.Priority
	logger      *log.Logger
	writer      *bufio.Writer
	destStream *os.File
	prefix string
	subloggers []*Logger
	initialized bool
}

// ParseLogLevel takes a string and returns a syslog.Priority according
// to the standard syslog string representation. Not case-sensitive.
func ParseLogLevel(level string) (syslog.Priority, error) {
	level = strings.ToLower(level)
	switch {
	case level == "emerg":
		return syslog.LOG_EMERG, nil
	case level == "alert":
		return syslog.LOG_ALERT, nil
	case level == "crit":
		return syslog.LOG_CRIT, nil
	case level == "err":
		return syslog.LOG_ERR, nil
	case level == "warning":
		return syslog.LOG_WARNING, nil
	case level == "notice":
		return syslog.LOG_NOTICE, nil
	case level == "info":
		return syslog.LOG_INFO, nil
	case level == "debug":
		return syslog.LOG_DEBUG, nil
	}
	return syslog.Priority(0), fmt.Errorf("Invalid log level: %s", level)
}

// Return a new Logger. logLevel is a syslog log level,
// subpackage is used to construct the log prefix, and dest is where to
// write the log to.
func NewLogger(logLevel syslog.Priority, subpackage string, dest *os.File) *Logger {
	logger := new(Logger)
	logger.logLevel = logLevel
	flags := log.Ldate | log.Ltime
	// If logging at DEBUG, include file paths and line numbers
	if logLevel == syslog.LOG_DEBUG {
		flags |= log.Lshortfile
	}
	logger.prefix = fmt.Sprintf("[%s] ", subpackage)
	logger.destStream = dest
	logger.writer = bufio.NewWriter(logger.destStream)
	logger.logger = log.New(logger.writer, logger.prefix, flags)
	logger.initialized = true
	return logger
}

// NewDefaultLogger returns a picolog.Logger initialized with workable
// defaults (outputs to stderr, prefix "default", priority DEBUG).
// Useful as a fallback when a logger hasn't been initialized.
func NewDefaultLogger() *Logger {
	return NewLogger(syslog.LOG_DEBUG, "default", os.Stderr)
}

// initializeDefaultLogger takes a (possibly nil) *Logger and allocates
// and assigns a default logger as returned by NewDefaultLogger.
func (l *Logger) initializeDefaultLogger() {
	defaultLogger := NewDefaultLogger()
	l = defaultLogger
}

// ensureInitialized checks if the initialized flag has been set for l,
// and if not initializes a default logger.
func (l *Logger) ensureInitialized() {
	if !l.initialized {
		l.initializeDefaultLogger()
	}
}

// NewSubLogger returns a Logger writing to the same stream, with a
// prefix constructed from the provided prefix and the parent Logger's
// prefix. Subloggers can be nested.
func (l *Logger) NewSubLogger(level syslog.Priority, prefix string) *Logger {
	subPrefix := fmt.Sprintf("%s[%s]", l.prefix, prefix)
	sub := NewLogger(level, subPrefix, l.destStream)
	l.subloggers = append(l.subloggers, sub)
	return sub
}

// Printf is the lowest-level output function of our Logger. Will use a
// default logger if l is not initialized.
func (l *Logger) Printf(format string, level syslog.Priority, v ...interface{}) {
	l.ensureInitialized()
	if level <= l.logLevel {
		msg := fmt.Sprintf(format, v...)
		// We use logger.Output rather than logger.Printf
		// so we can pass a custom calldepth for file
		// {path,line}-resolution purposes (the default of 2
		// is only useful when using the Logger type directly).
		l.logger.Output(3, msg)
		l.writer.Flush()
	}
}

// Debugf logs one printf-formatted message at LOG_DEBUG.
func (l *Logger) Debugf(format string, v ...interface{}) {
	l.Printf(format, syslog.LOG_DEBUG, v...)
}

// Errorf logs one printf-formatted message at LOG_ERR.
func (l *Logger) Errorf(format string, v ...interface{}) {
	l.Printf(format, syslog.LOG_ERR, v...)
}

// Warningf logs one printf-formatted message at LOG_WARNING.
func (l *Logger) Warningf(format string, v ...interface{}) {
	l.Printf(format, syslog.LOG_WARNING, v...)
}

// Fatalf logs one printf-formatted message at LOG_CRIT, and then exits
// with an error code.
func (l *Logger) Fatalf(format string, v ...interface{}) {
	l.Printf(format, syslog.LOG_CRIT, v...)
	os.Exit(1)
}

// Infof logs one printf-formatted message at LOG_INFO.
func (l *Logger) Infof(format string, v ...interface{}) {
	l.Printf(format, syslog.LOG_INFO, v...)
}
<|endoftext|>


<commit_before>package wc

import (
	"fmt"
	"testing"
)

var testCases = []struct {
	description string
	input       string
	output      Histogram
}{
	{
		description: "a single word",
		input:       "word",
		output:      Histogram{"word": 1},
	},
	{
		description: "one of each",
		input:       "one of each",
		output:      Histogram{"one": 1, "of": 1, "each": 1},
	},
	{
		description: "multiple occurrences",
		input:       "one fish two fish red fish blue fish",
		output:      Histogram{"one": 1, "fish": 4, "two": 1, "red": 1, "blue": 1},
	},
	{
		description: "ignore punctuation",
		input:       "car : carpet as java : javascript!!&@$%^&",
		output:      Histogram{"car": 1, "carpet": 1, "as": 1, "java": 1, "javascript": 1},
	},
	{
		description: "including numbers",
		input:       "testing, 1, 2 testing",
		output:      Histogram{"testing": 2, "1": 1, "2": 1},
	},
	{
		description: "normalises case",
		input:       "go Go GO",
		output:      Histogram{"go": 3},
	},
}

func TestWordCount(t *testing.T) {
	for _, tt := range testCases {
		expected := fmt.Sprintf("%v", tt.output)
		actual := fmt.Sprintf("%v", WordCount(tt.input))

		if expected != actual {
			t.Fatalf("%s\n\tExpected: %v\n\tGot: %v", tt.description, expected, actual)
		} else {
			t.Logf("PASS: %s - WordCount(%s)", tt.description, tt.input)
		}
	}
}
<commit_msg>You can't compare the two maps with !=, because it compares the string representation of the maps, so it will work only randomly.<commit_after>package wc

import (
	"fmt"
	"reflect"
	"testing"
)

var testCases = []struct {
	description string
	input       string
	output      Histogram
}{
	{
		description: "a single word",
		input:       "word",
		output:      Histogram{"word": 1},
	},
	{
		description: "one of each",
		input:       "one of each",
		output:      Histogram{"one": 1, "of": 1, "each": 1},
	},
	{
		description: "multiple occurrences",
		input:       "one fish two fish red fish blue fish",
		output:      Histogram{"one": 1, "fish": 4, "two": 1, "red": 1, "blue": 1},
	},
	{
		description: "ignore punctuation",
		input:       "car : carpet as java : javascript!!&@$%^&",
		output:      Histogram{"car": 1, "carpet": 1, "as": 1, "java": 1, "javascript": 1},
	},
	{
		description: "including numbers",
		input:       "testing, 1, 2 testing",
		output:      Histogram{"testing": 2, "1": 1, "2": 1},
	},
	{
		description: "normalises case",
		input:       "go Go GO",
		output:      Histogram{"go": 3},
	},
}

func TestWordCount(t *testing.T) {
	for _, tt := range testCases {
		expected := fmt.Sprintf("%v", tt.output)
		actual := fmt.Sprintf("%v", WordCount(tt.input))

		if !reflect.DeepEqual(tt.output, WordCount(tt.input)) {
			t.Fatalf("%s\n\tExpected: %v\n\tGot: %v", tt.description, expected, actual)
		} else {
			t.Logf("PASS: %s - WordCount(%s)", tt.description, tt.input)
		}
	}
}
<|endoftext|>


<commit_before>package main

import (
	"container/heap"
	"fmt"
	"github.com/jessevdk/go-flags"
	"github.com/tj/go-dropbox"
	"golang.org/x/text/unicode/norm"
	"math"
	"os"
	"path/filepath"
	"runtime"
	"strings"
	"sync"
	"time"
)

/*
TODO
- Performance improvements:
	- Test if buffered channels improve performance in the parallel local file processing
	- Profile to find other bottlenecks?
	- Could printing progress for each local file result slow things down? (When processing lots of small files)
- Print I/O usage? i.e. how many MB/s are we processing
- Clean up output formatting
- Ignore more file names in skipLocalFile - see https://www.dropbox.com/help/syncing-uploads/files-not-syncing
- Do a real retry + backoff for Dropbox API errors (do we have access to the Retry-After header?)
*/

// File stores the result of either Dropbox API or local file listing
type File struct {
	Path        string
	ContentHash string
}

// FileError records a local file that could not be read due to an error
type FileError struct {
	Path  string
	Error error
}

// FileHeap is a list of Files sorted by path
type FileHeap []*File

func (h FileHeap) Len() int           { return len(h) }
func (h FileHeap) Less(i, j int) bool { return h[i].Path < h[j].Path }
func (h FileHeap) Swap(i, j int)      { h[i], h[j] = h[j], h[i] }

// Push a File onto the heap
func (h *FileHeap) Push(x interface{}) {
	// Push and Pop use pointer receivers because they modify the slice's length,
	// not just its contents.
	*h = append(*h, x.(*File))
}

// Pop a File off the heap
func (h *FileHeap) Pop() interface{} {
	old := *h
	n := len(old)
	x := old[n-1]
	*h = old[0 : n-1]
	return x
}

// PopOrNil pops a File off the heap or returns nil if there's nothing left
func (h *FileHeap) PopOrNil() *File {
	if h.Len() > 0 {
		return heap.Pop(h).(*File)
	}
	return nil
}

// ManifestComparison records the relative paths that differ between remote and
// local versions of a directory
type ManifestComparison struct {
	OnlyRemote      []string
	OnlyLocal       []string
	ContentMismatch []string
	Errored         []*FileError
	Matches         int
	Misses          int
}

type progressType int

const (
	remoteProgress progressType = iota
	localProgress
	errorProgress
)

type scanProgressUpdate struct {
	Type  progressType
	Count int
}

func main() {
	token := os.Getenv("DROPBOX_ACCESS_TOKEN")
	if token == "" {
		fmt.Fprintln(os.Stderr, "Missing Dropbox OAuth token! Please set the DROPBOX_ACCESS_TOKEN environment variable.")
		os.Exit(1)
	}

	var opts struct {
		Verbose          bool   `short:"v" long:"verbose" description:"Show verbose debug information"`
		RemoteRoot       string `short:"r" long:"remote" description:"Directory in Dropbox to verify" default:"/"`
		LocalRoot        string `short:"l" long:"local" description:"Local directory to compare to Dropbox contents" default:"."`
		CheckContentHash bool   `long:"check" description:"Check content hash of local files"`
		WorkerCount      int    `short:"w" long:"workers" description:"Number of worker threads to use (defaults to 8)" default:"8"`
	}

	_, err := flags.Parse(&opts)
	if err != nil {
		fmt.Fprintln(os.Stderr, err.Error())
		os.Exit(1)
	}

	// Dropbox API uses empty string for root, but for figuring out relative
	// paths of the returned entries it's easier to use "/". Conversion is
	// handled before the API call.
	if opts.RemoteRoot == "" {
		opts.RemoteRoot = "/"
	}
	if opts.RemoteRoot[0] != '/' {
		opts.RemoteRoot = "/" + opts.RemoteRoot
	}

	localRoot, _ := filepath.Abs(opts.LocalRoot)

	dbxClient := dropbox.New(dropbox.NewConfig(token))

	fmt.Printf("Comparing Dropbox directory \"%v\" to local directory \"%v\"\n", opts.RemoteRoot, localRoot)
	if opts.CheckContentHash {
		fmt.Println("Checking content hashes.")
	}
	fmt.Println("")

	progressChan := make(chan *scanProgressUpdate)
	var wg sync.WaitGroup
	wg.Add(2)

	var dropboxManifest *FileHeap
	var dropboxErr error
	go func() {
		dropboxManifest, dropboxErr = getDropboxManifest(progressChan, dbxClient, opts.RemoteRoot)
		wg.Done()
	}()

	var localManifest *FileHeap
	var errored []*FileError
	var localErr error
	go func() {
		localManifest, errored, localErr = getLocalManifest(progressChan, localRoot, opts.CheckContentHash, opts.WorkerCount)
		wg.Done()
	}()

	go func() {
		remoteCount := 0
		localCount := 0
		errorCount := 0
		for update := range progressChan {
			switch update.Type {
			case remoteProgress:
				remoteCount = update.Count
			case localProgress:
				localCount = update.Count
			case errorProgress:
				errorCount = update.Count
			}

			if opts.Verbose {
				fmt.Fprintf(os.Stderr, "Scanning: %d (remote) %d (local) %d (errored)\r", remoteCount, localCount, errorCount)
			}
		}
		fmt.Fprintf(os.Stderr, "\n")
	}()

	// wait until remote and local scans are complete, then close progress reporting channel
	wg.Wait()
	close(progressChan)
	fmt.Printf("\nGenerated manifests for %d remote files, %d local files, with %d local errors\n\n", dropboxManifest.Len(), localManifest.Len(), len(errored))

	// check for fatal errors
	if dropboxErr != nil {
		panic(dropboxErr)
	}
	if localErr != nil {
		panic(localErr)
	}

	manifestComparison := compareManifests(dropboxManifest, localManifest, errored)

	fmt.Println("")

	printFileList(manifestComparison.OnlyRemote, "Files only in remote")
	printFileList(manifestComparison.OnlyLocal, "Files only in local")
	printFileList(manifestComparison.ContentMismatch, "Files whose contents don't match")

	fmt.Printf("Errored: %d\n\n", len(manifestComparison.Errored))
	if len(manifestComparison.Errored) > 0 {
		for _, rec := range manifestComparison.Errored {
			fmt.Printf("%s: %s\n", rec.Path, rec.Error)
		}
		if len(manifestComparison.Errored) > 0 {
			fmt.Print("\n\n")
		}
	}

	total := manifestComparison.Matches + manifestComparison.Misses
	fmt.Println("SUMMARY:")
	fmt.Printf("Files matched: %d/%d\n", manifestComparison.Matches, total)
	fmt.Printf("Files not matched: %d/%d\n", manifestComparison.Misses, total)
}

func getDropboxManifest(progressChan chan<- *scanProgressUpdate, dbxClient *dropbox.Client, rootPath string) (manifest *FileHeap, err error) {
	manifest = &FileHeap{}
	heap.Init(manifest)
	cursor := ""
	keepGoing := true

	for keepGoing {
		var resp *dropbox.ListFolderOutput
		if cursor != "" {
			arg := &dropbox.ListFolderContinueInput{Cursor: cursor}
			resp, err = dbxClient.Files.ListFolderContinue(arg)
		} else {
			apiPath := rootPath
			if apiPath == "/" {
				apiPath = ""
			}
			arg := &dropbox.ListFolderInput{
				Path:             apiPath,
				Recursive:        true,
				IncludeMediaInfo: false,
				IncludeDeleted:   false,
			}
			resp, err = dbxClient.Files.ListFolder(arg)
		}
		if err != nil {
			if strings.HasPrefix(err.Error(), "too_many_requests") {
				fmt.Fprint(os.Stderr, "Dropbox returned too many requests error, sleeping 60 seconds...\n")
				fmt.Fprintf(os.Stderr, "Error: %v\n", err)
				fmt.Fprintf(os.Stderr, "Response: %v\n", resp)
				time.Sleep(60 * time.Second)
				continue
			}
			return
		}
		for _, entry := range resp.Entries {
			if entry.Tag == "file" {

				var relPath string
				relPath, err = normalizePath(rootPath, entry.PathLower)
				if err != nil {
					return
				}
				heap.Push(manifest, &File{
					Path:        relPath,
					ContentHash: entry.ContentHash,
				})
			}
		}

		cursor = resp.Cursor
		keepGoing = resp.HasMore

		progressChan <- &scanProgressUpdate{Type: remoteProgress, Count: manifest.Len()}
	}

	return
}

func getLocalManifest(progressChan chan<- *scanProgressUpdate, localRoot string, contentHash bool, workerCount int) (manifest *FileHeap, errored []*FileError, err error) {
	localRootLowercase := strings.ToLower(localRoot)
	manifest = &FileHeap{}
	heap.Init(manifest)
	if workerCount <= 0 {
		workerCount = int(math.Max(1, float64(runtime.NumCPU())))
	}
	processChan := make(chan string)
	resultChan := make(chan *File)
	errorChan := make(chan *FileError)
	var wg sync.WaitGroup

	for i := 0; i < workerCount; i++ {
		// spin up workers
		wg.Add(1)
		go handleLocalFile(localRootLowercase, contentHash, processChan, resultChan, errorChan, &wg)
	}

	// walk in separate goroutine so that sends to errorChan don't block
	go func() {
		filepath.Walk(localRoot, func(entryPath string, info os.FileInfo, err error) error {
			if err != nil {
				errorChan <- &FileError{Path: entryPath, Error: err}
				return nil
			}

			if info.Mode().IsDir() && skipLocalDir(entryPath) {
				return filepath.SkipDir
			}

			if info.Mode().IsRegular() && !skipLocalFile(entryPath) {
				processChan <- entryPath
			}

			return nil
		})

		close(processChan)
	}()

	// Once processing goroutines are done, close result and error channels to indicate no more results streaming in
	go func() {
		wg.Wait()
		close(resultChan)
		close(errorChan)
	}()

	for {
		select {
		case result, ok := <-resultChan:
			if ok {
				heap.Push(manifest, result)
				progressChan <- &scanProgressUpdate{Type: localProgress, Count: manifest.Len()}
			} else {
				resultChan = nil
			}

		case e, ok := <-errorChan:
			if ok {
				errored = append(errored, e)
				progressChan <- &scanProgressUpdate{Type: errorProgress, Count: len(errored)}
			} else {
				errorChan = nil
			}
		}

		if resultChan == nil && errorChan == nil {
			break
		}
	}

	return
}

// fill in args etc
func handleLocalFile(localRootLowercase string, contentHash bool, processChan <-chan string, resultChan chan<- *File, errorChan chan<- *FileError, wg *sync.WaitGroup) {
	for entryPath := range processChan {

		relPath, err := normalizePath(localRootLowercase, strings.ToLower(entryPath))
		if err != nil {
			errorChan <- &FileError{Path: entryPath, Error: err}
			continue
		}

		hash := ""
		if contentHash {
			hash, err = dropbox.FileContentHash(entryPath)
			if err != nil {
				errorChan <- &FileError{Path: relPath, Error: err}
				continue
			}
		}

		resultChan <- &File{
			Path:        relPath,
			ContentHash: hash,
		}
	}
	wg.Done()
}

func normalizePath(root string, entryPath string) (string, error) {
	relPath, err := filepath.Rel(root, entryPath)
	if err != nil {
		return "", err
	}
	if relPath[0:3] == "../" {
		// try lowercase root instead
		relPath, err = filepath.Rel(strings.ToLower(root), entryPath)
		if err != nil {
			return "", err
		}
	}

	// Normalize Unicode combining characters
	relPath = norm.NFC.String(relPath)
	return relPath, nil
}

func skipLocalFile(path string) bool {
	if filepath.Base(path) == ".DS_Store" {
		return true
	}
	return false
}

func skipLocalDir(path string) bool {
	if filepath.Base(path) == "@eaDir" {
		return true
	}
	return false
}

func compareManifests(remoteManifest, localManifest *FileHeap, errored []*FileError) *ManifestComparison {
	// 1. Pop a path off both remote and local manifests.
	// 2. While remote & local are both not nil:
	//    Compare remote & local:
	//    a. If local is nil or local > remote, this file is only in remote. Record and pop remote again.
	//    b. If remote is nil or local < remote, this file is only in local. Record and pop local again.
	//    c. If local == remote, check for content mismatch. Record if necessary and pop both again.
	comparison := &ManifestComparison{Errored: errored}
	local := localManifest.PopOrNil()
	remote := remoteManifest.PopOrNil()
	for local != nil || remote != nil {
		if local == nil {
			comparison.OnlyRemote = append(comparison.OnlyRemote, remote.Path)
			comparison.Misses++
			remote = remoteManifest.PopOrNil()
		} else if remote == nil {
			comparison.OnlyLocal = append(comparison.OnlyLocal, local.Path)
			comparison.Misses++
			local = localManifest.PopOrNil()
		} else if local.Path > remote.Path {
			comparison.OnlyRemote = append(comparison.OnlyRemote, remote.Path)
			comparison.Misses++
			remote = remoteManifest.PopOrNil()
		} else if local.Path < remote.Path {
			comparison.OnlyLocal = append(comparison.OnlyLocal, local.Path)
			comparison.Misses++
			local = localManifest.PopOrNil()
		} else {
			// this must mean that remote.Path == local.Path
			if compareFileContents(remote, local) {
				comparison.Matches++
			} else {
				comparison.ContentMismatch = append(comparison.ContentMismatch, local.Path)
				comparison.Misses++
			}
			local = localManifest.PopOrNil()
			remote = remoteManifest.PopOrNil()
		}
	}
	return comparison
}

func compareFileContents(remote, local *File) bool {
	if remote.ContentHash == "" || local.ContentHash == "" {
		// Missing content hash for one of the files, possibly intentionally,
		// so can't compare. Assume that presence of both is enough to
		// validate.
		return true
	}
	return remote.ContentHash == local.ContentHash
}

func printFileList(files []string, description string) {
	fmt.Printf("%s: %d\n\n", description, len(files))
	for _, path := range files {
		fmt.Println(path)
	}
	if len(files) > 0 {
		fmt.Print("\n\n")
	}
}
<commit_msg>Better Dropbox error formatting<commit_after>package main

import (
	"container/heap"
	"fmt"
	"github.com/jessevdk/go-flags"
	"github.com/tj/go-dropbox"
	"golang.org/x/text/unicode/norm"
	"math"
	"os"
	"path/filepath"
	"runtime"
	"strings"
	"sync"
	"time"
)

/*
TODO
- Performance improvements:
	- Test if buffered channels improve performance in the parallel local file processing
	- Profile to find other bottlenecks?
	- Could printing progress for each local file result slow things down? (When processing lots of small files)
- Print I/O usage? i.e. how many MB/s are we processing
- Clean up output formatting
- Ignore more file names in skipLocalFile - see https://www.dropbox.com/help/syncing-uploads/files-not-syncing
- Do a real retry + backoff for Dropbox API errors (do we have access to the Retry-After header?)
*/

// File stores the result of either Dropbox API or local file listing
type File struct {
	Path        string
	ContentHash string
}

// FileError records a local file that could not be read due to an error
type FileError struct {
	Path  string
	Error error
}

// FileHeap is a list of Files sorted by path
type FileHeap []*File

func (h FileHeap) Len() int           { return len(h) }
func (h FileHeap) Less(i, j int) bool { return h[i].Path < h[j].Path }
func (h FileHeap) Swap(i, j int)      { h[i], h[j] = h[j], h[i] }

// Push a File onto the heap
func (h *FileHeap) Push(x interface{}) {
	// Push and Pop use pointer receivers because they modify the slice's length,
	// not just its contents.
	*h = append(*h, x.(*File))
}

// Pop a File off the heap
func (h *FileHeap) Pop() interface{} {
	old := *h
	n := len(old)
	x := old[n-1]
	*h = old[0 : n-1]
	return x
}

// PopOrNil pops a File off the heap or returns nil if there's nothing left
func (h *FileHeap) PopOrNil() *File {
	if h.Len() > 0 {
		return heap.Pop(h).(*File)
	}
	return nil
}

// ManifestComparison records the relative paths that differ between remote and
// local versions of a directory
type ManifestComparison struct {
	OnlyRemote      []string
	OnlyLocal       []string
	ContentMismatch []string
	Errored         []*FileError
	Matches         int
	Misses          int
}

type progressType int

const (
	remoteProgress progressType = iota
	localProgress
	errorProgress
)

type scanProgressUpdate struct {
	Type  progressType
	Count int
}

func main() {
	token := os.Getenv("DROPBOX_ACCESS_TOKEN")
	if token == "" {
		fmt.Fprintln(os.Stderr, "Missing Dropbox OAuth token! Please set the DROPBOX_ACCESS_TOKEN environment variable.")
		os.Exit(1)
	}

	var opts struct {
		Verbose          bool   `short:"v" long:"verbose" description:"Show verbose debug information"`
		RemoteRoot       string `short:"r" long:"remote" description:"Directory in Dropbox to verify" default:"/"`
		LocalRoot        string `short:"l" long:"local" description:"Local directory to compare to Dropbox contents" default:"."`
		CheckContentHash bool   `long:"check" description:"Check content hash of local files"`
		WorkerCount      int    `short:"w" long:"workers" description:"Number of worker threads to use (defaults to 8)" default:"8"`
	}

	_, err := flags.Parse(&opts)
	if err != nil {
		fmt.Fprintln(os.Stderr, err.Error())
		os.Exit(1)
	}

	// Dropbox API uses empty string for root, but for figuring out relative
	// paths of the returned entries it's easier to use "/". Conversion is
	// handled before the API call.
	if opts.RemoteRoot == "" {
		opts.RemoteRoot = "/"
	}
	if opts.RemoteRoot[0] != '/' {
		opts.RemoteRoot = "/" + opts.RemoteRoot
	}

	localRoot, _ := filepath.Abs(opts.LocalRoot)

	dbxClient := dropbox.New(dropbox.NewConfig(token))

	fmt.Printf("Comparing Dropbox directory \"%v\" to local directory \"%v\"\n", opts.RemoteRoot, localRoot)
	if opts.CheckContentHash {
		fmt.Println("Checking content hashes.")
	}
	fmt.Println("")

	progressChan := make(chan *scanProgressUpdate)
	var wg sync.WaitGroup
	wg.Add(2)

	var dropboxManifest *FileHeap
	var dropboxErr error
	go func() {
		dropboxManifest, dropboxErr = getDropboxManifest(progressChan, dbxClient, opts.RemoteRoot)
		wg.Done()
	}()

	var localManifest *FileHeap
	var errored []*FileError
	var localErr error
	go func() {
		localManifest, errored, localErr = getLocalManifest(progressChan, localRoot, opts.CheckContentHash, opts.WorkerCount)
		wg.Done()
	}()

	go func() {
		remoteCount := 0
		localCount := 0
		errorCount := 0
		for update := range progressChan {
			switch update.Type {
			case remoteProgress:
				remoteCount = update.Count
			case localProgress:
				localCount = update.Count
			case errorProgress:
				errorCount = update.Count
			}

			if opts.Verbose {
				fmt.Fprintf(os.Stderr, "Scanning: %d (remote) %d (local) %d (errored)\r", remoteCount, localCount, errorCount)
			}
		}
		fmt.Fprintf(os.Stderr, "\n")
	}()

	// wait until remote and local scans are complete, then close progress reporting channel
	wg.Wait()
	close(progressChan)
	fmt.Printf("\nGenerated manifests for %d remote files, %d local files, with %d local errors\n\n", dropboxManifest.Len(), localManifest.Len(), len(errored))

	// check for fatal errors
	if dropboxErr != nil {
		panic(dropboxErr)
	}
	if localErr != nil {
		panic(localErr)
	}

	manifestComparison := compareManifests(dropboxManifest, localManifest, errored)

	fmt.Println("")

	printFileList(manifestComparison.OnlyRemote, "Files only in remote")
	printFileList(manifestComparison.OnlyLocal, "Files only in local")
	printFileList(manifestComparison.ContentMismatch, "Files whose contents don't match")

	fmt.Printf("Errored: %d\n\n", len(manifestComparison.Errored))
	if len(manifestComparison.Errored) > 0 {
		for _, rec := range manifestComparison.Errored {
			fmt.Printf("%s: %s\n", rec.Path, rec.Error)
		}
		if len(manifestComparison.Errored) > 0 {
			fmt.Print("\n\n")
		}
	}

	total := manifestComparison.Matches + manifestComparison.Misses
	fmt.Println("SUMMARY:")
	fmt.Printf("Files matched: %d/%d\n", manifestComparison.Matches, total)
	fmt.Printf("Files not matched: %d/%d\n", manifestComparison.Misses, total)
}

func getDropboxManifest(progressChan chan<- *scanProgressUpdate, dbxClient *dropbox.Client, rootPath string) (manifest *FileHeap, err error) {
	manifest = &FileHeap{}
	heap.Init(manifest)
	cursor := ""
	keepGoing := true

	for keepGoing {
		var resp *dropbox.ListFolderOutput
		if cursor != "" {
			arg := &dropbox.ListFolderContinueInput{Cursor: cursor}
			resp, err = dbxClient.Files.ListFolderContinue(arg)
		} else {
			apiPath := rootPath
			if apiPath == "/" {
				apiPath = ""
			}
			arg := &dropbox.ListFolderInput{
				Path:             apiPath,
				Recursive:        true,
				IncludeMediaInfo: false,
				IncludeDeleted:   false,
			}
			resp, err = dbxClient.Files.ListFolder(arg)
		}
		if err != nil {
			if strings.HasPrefix(err.Error(), "too_many_requests") {
				fmt.Fprintf(os.Stderr, "\n[%s] Dropbox returned too many requests error, sleeping 60 seconds\n", time.Now().Format("15:04:05"))
				// fmt.Fprintf(os.Stderr, "Error: %v\n", err)
				// fmt.Fprintf(os.Stderr, "Response: %v\n", resp)
				time.Sleep(60 * time.Second)
				continue
			}
			return
		}
		for _, entry := range resp.Entries {
			if entry.Tag == "file" {

				var relPath string
				relPath, err = normalizePath(rootPath, entry.PathLower)
				if err != nil {
					return
				}
				heap.Push(manifest, &File{
					Path:        relPath,
					ContentHash: entry.ContentHash,
				})
			}
		}

		cursor = resp.Cursor
		keepGoing = resp.HasMore

		progressChan <- &scanProgressUpdate{Type: remoteProgress, Count: manifest.Len()}
	}

	return
}

func getLocalManifest(progressChan chan<- *scanProgressUpdate, localRoot string, contentHash bool, workerCount int) (manifest *FileHeap, errored []*FileError, err error) {
	localRootLowercase := strings.ToLower(localRoot)
	manifest = &FileHeap{}
	heap.Init(manifest)
	if workerCount <= 0 {
		workerCount = int(math.Max(1, float64(runtime.NumCPU())))
	}
	processChan := make(chan string)
	resultChan := make(chan *File)
	errorChan := make(chan *FileError)
	var wg sync.WaitGroup

	for i := 0; i < workerCount; i++ {
		// spin up workers
		wg.Add(1)
		go handleLocalFile(localRootLowercase, contentHash, processChan, resultChan, errorChan, &wg)
	}

	// walk in separate goroutine so that sends to errorChan don't block
	go func() {
		filepath.Walk(localRoot, func(entryPath string, info os.FileInfo, err error) error {
			if err != nil {
				errorChan <- &FileError{Path: entryPath, Error: err}
				return nil
			}

			if info.Mode().IsDir() && skipLocalDir(entryPath) {
				return filepath.SkipDir
			}

			if info.Mode().IsRegular() && !skipLocalFile(entryPath) {
				processChan <- entryPath
			}

			return nil
		})

		close(processChan)
	}()

	// Once processing goroutines are done, close result and error channels to indicate no more results streaming in
	go func() {
		wg.Wait()
		close(resultChan)
		close(errorChan)
	}()

	for {
		select {
		case result, ok := <-resultChan:
			if ok {
				heap.Push(manifest, result)
				progressChan <- &scanProgressUpdate{Type: localProgress, Count: manifest.Len()}
			} else {
				resultChan = nil
			}

		case e, ok := <-errorChan:
			if ok {
				errored = append(errored, e)
				progressChan <- &scanProgressUpdate{Type: errorProgress, Count: len(errored)}
			} else {
				errorChan = nil
			}
		}

		if resultChan == nil && errorChan == nil {
			break
		}
	}

	return
}

// fill in args etc
func handleLocalFile(localRootLowercase string, contentHash bool, processChan <-chan string, resultChan chan<- *File, errorChan chan<- *FileError, wg *sync.WaitGroup) {
	for entryPath := range processChan {

		relPath, err := normalizePath(localRootLowercase, strings.ToLower(entryPath))
		if err != nil {
			errorChan <- &FileError{Path: entryPath, Error: err}
			continue
		}

		hash := ""
		if contentHash {
			hash, err = dropbox.FileContentHash(entryPath)
			if err != nil {
				errorChan <- &FileError{Path: relPath, Error: err}
				continue
			}
		}

		resultChan <- &File{
			Path:        relPath,
			ContentHash: hash,
		}
	}
	wg.Done()
}

func normalizePath(root string, entryPath string) (string, error) {
	relPath, err := filepath.Rel(root, entryPath)
	if err != nil {
		return "", err
	}
	if relPath[0:3] == "../" {
		// try lowercase root instead
		relPath, err = filepath.Rel(strings.ToLower(root), entryPath)
		if err != nil {
			return "", err
		}
	}

	// Normalize Unicode combining characters
	relPath = norm.NFC.String(relPath)
	return relPath, nil
}

func skipLocalFile(path string) bool {
	if filepath.Base(path) == ".DS_Store" {
		return true
	}
	return false
}

func skipLocalDir(path string) bool {
	if filepath.Base(path) == "@eaDir" {
		return true
	}
	return false
}

func compareManifests(remoteManifest, localManifest *FileHeap, errored []*FileError) *ManifestComparison {
	// 1. Pop a path off both remote and local manifests.
	// 2. While remote & local are both not nil:
	//    Compare remote & local:
	//    a. If local is nil or local > remote, this file is only in remote. Record and pop remote again.
	//    b. If remote is nil or local < remote, this file is only in local. Record and pop local again.
	//    c. If local == remote, check for content mismatch. Record if necessary and pop both again.
	comparison := &ManifestComparison{Errored: errored}
	local := localManifest.PopOrNil()
	remote := remoteManifest.PopOrNil()
	for local != nil || remote != nil {
		if local == nil {
			comparison.OnlyRemote = append(comparison.OnlyRemote, remote.Path)
			comparison.Misses++
			remote = remoteManifest.PopOrNil()
		} else if remote == nil {
			comparison.OnlyLocal = append(comparison.OnlyLocal, local.Path)
			comparison.Misses++
			local = localManifest.PopOrNil()
		} else if local.Path > remote.Path {
			comparison.OnlyRemote = append(comparison.OnlyRemote, remote.Path)
			comparison.Misses++
			remote = remoteManifest.PopOrNil()
		} else if local.Path < remote.Path {
			comparison.OnlyLocal = append(comparison.OnlyLocal, local.Path)
			comparison.Misses++
			local = localManifest.PopOrNil()
		} else {
			// this must mean that remote.Path == local.Path
			if compareFileContents(remote, local) {
				comparison.Matches++
			} else {
				comparison.ContentMismatch = append(comparison.ContentMismatch, local.Path)
				comparison.Misses++
			}
			local = localManifest.PopOrNil()
			remote = remoteManifest.PopOrNil()
		}
	}
	return comparison
}

func compareFileContents(remote, local *File) bool {
	if remote.ContentHash == "" || local.ContentHash == "" {
		// Missing content hash for one of the files, possibly intentionally,
		// so can't compare. Assume that presence of both is enough to
		// validate.
		return true
	}
	return remote.ContentHash == local.ContentHash
}

func printFileList(files []string, description string) {
	fmt.Printf("%s: %d\n\n", description, len(files))
	for _, path := range files {
		fmt.Println(path)
	}
	if len(files) > 0 {
		fmt.Print("\n\n")
	}
}
<|endoftext|>