<commit_before>// Copyright 2012 Google, Inc. All rights reserved.
//
// Use of this source code is governed by a BSD-style license
// that can be found in the LICENSE file in the root of the source
// tree.

/*
Package gopacket provides packet decoding for the Go language.

gopacket contains many sub-packages with additional functionality you may find
useful, including:

 * layers: You'll probably use this every time.  This contains of the logic
     built into gopacket for decoding packet protocols.  Note that all example
     code below assumes that you have imported both gopacket and
     gopacket/layers.
 * pcap: C bindings to use libpcap to read packets off the wire.
 * pfring: C bindings to use PF_RING to read packets off the wire.
 * afpacket: C bindings for Linux's AF_PACKET to read packets off the wire.
 * tcpassembly: TCP stream reassembly

Also, if you're looking to dive right into code, see the examples subdirectory
for numerous simple binaries built using gopacket libraries.

Basic Usage

gopacket takes in packet data as a []byte and decodes it into a packet with
a non-zero number of "layers".  Each layer corresponds to a protocol
within the bytes.  Once a packet has been decoded, the layers of the packet
can be requested from the packet.

 // Decode a packet
 packet := gopacket.NewPacket(myPacketData, layers.LayerTypeEthernet, gopacket.Default)
 // Get the TCP layer from this packet
 if tcpLayer := packet.Layer(layers.LayerTypeTCP); tcpLayer != nil {
   fmt.Println("This is a TCP packet!")
   // Get actual TCP data from this layer
   tcp, _ := tcpLayer.(*layers.TCP)
   fmt.Printf("From src port %d to dst port %d\n", tcp.SrcPort, tcp.DstPort)
 }
 // Iterate over all layers, printing out each layer type
 for _, layer := range packet.Layers() {
   fmt.Println("PACKET LAYER:", layer.LayerType())
 }

Packets can be decoded from a number of starting points.  Many of our base
types implement Decoder, which allow us to decode packets for which
we don't have full data.

 // Decode an ethernet packet
 ethP := gopacket.NewPacket(p1, layers.LayerTypeEthernet, gopacket.Default)
 // Decode an IPv6 header and everything it contains
 ipP := gopacket.NewPacket(p2, layers.LayerTypeIPv6, gopacket.Default)
 // Decode a TCP header and its payload
 tcpP := gopacket.NewPacket(p3, layers.LayerTypeTCP, gopacket.Default)

Reading Packets From A Source

Most of the time, you won't just have a []byte of packet data lying around.
Instead, you'll want to read packets in from somewhere (file, interface, etc)
and process them.  To do that, you'll want to build a PacketSource.

First, you'll need to construct an object that implements the PacketDataSource
interface.  There are implementations of this interface bundled with gopacket
in the gopacket/pcap and gopacket/pfring subpackages... see their documentation
for more information on their usage.  Once you have a PacketDataSource, you can
pass it into NewPacketSource, along with a Decoder of your choice, to create
a PacketSource.

Once you have a PacketSource, you can read packets from it in multiple ways.
See the docs for PacketSource for more details.  The easiest method is the
Packets function, which returns a channel, then asynchronously writes new
packets into that channel, closing the channel if the packetSource hits an
end-of-file.

  packetSource := ...  // construct using pcap or pfring
  for packet := range packetSource.Packets() {
    handlePacket(packet)  // do something with each packet
  }

You can change the decoding options of the packetSource by setting fields in
packetSource.DecodeOptions... see the following sections for more details.

Lazy Decoding

gopacket optionally decodes packet data lazily, meaning it
only decodes a packet layer when it needs to handle a function call.

 // Create a packet, but don't actually decode anything yet
 packet := gopacket.NewPacket(myPacketData, layers.LayerTypeEthernet, gopacket.Lazy)
 // Now, decode the packet up to the first IPv4 layer found but no further.
 // If no IPv4 layer was found, the whole packet will be decoded looking for
 // it.
 ip4 := packet.Layer(layers.LayerTypeIPv4)
 // Decode all layers and return them.  The layers up to the first IPv4 layer
 // are already decoded, and will not require decoding a second time.
 layers := packet.Layers()

Lazily-decoded packets are not concurrency-safe.  Since layers have not all been
decoded, each call to Layer() or Layers() has the potential to mutate the packet
in order to decode the next layer.  If a packet is used
in multiple goroutines concurrently, don't use gopacket.Lazy.  Then gopacket
will decode the packet fully, and all future function calls won't mutate the
object.

NoCopy Decoding

By default, gopacket will copy the slice passed to NewPacket and store the
copy within the packet, so future mutations to the bytes underlying the slice
don't affect the packet and its layers.  If you can guarantee that the
underlying slice bytes won't be changed, you can use NoCopy to tell
gopacket.NewPacket, and it'll use the passed-in slice itself.

 // This channel returns new byte slices, each of which points to a new
 // memory location that's guaranteed immutable for the duration of the
 // packet.
 for data := range myByteSliceChannel {
   p := gopacket.NewPacket(data, layers.LayerTypeEthernet, gopacket.NoCopy)
   doSomethingWithPacket(p)
 }

The fastest method of decoding is to use both Lazy and NoCopy, but note from
the many caveats above that for some implementations either or both may be
dangerous.

Pointers To Known Layers

During decoding, certain layers are stored in the packet as well-known
layer types.  For example, IPv4 and IPv6 are both considered NetworkLayer
layers, while TCP and UDP are both TransportLayer layers.  We support 4
layers, corresponding to the 4 layers of the TCP/IP layering scheme (roughly
anagalous to layers 2, 3, 4, and 7 of the OSI model).  To access these,
you can use the packet.LinkLayer, packet.NetworkLayer,
packet.TransportLayer, and packet.ApplicationLayer functions.  Each of
these functions returns a corresponding interface
(gopacket.{Link,Network,Transport,Application}Layer).  The first three
provide methods for getting src/dst addresses for that particular layer,
while the final layer provides a Payload function to get payload data.
This is helpful, for example, to get payloads for all packets regardless
of their underlying data type:

 // Get packets from some source
 for packet := range someSource {
   if app := packet.ApplicationLayer(); app != nil {
     if strings.Contains(string(app.Payload()), "magic string") {
       fmt.Println("Found magic string in a packet!")
     }
   }
 }

A particularly useful layer is ErrorLayer, which is set whenever there's
an error parsing part of the packet.

 packet := gopacket.NewPacket(myPacketData, layers.LayerTypeEthernet, gopacket.Default)
 if err := packet.ErrorLayer(); err != nil {
   fmt.Println("Error decoding some part of the packet:", err)
 }

Note that we don't return an error from NewPacket because we may have decoded
a number of layers successfully before running into our erroneous layer.  You
may still be able to get your Ethernet and IPv4 layers correctly, even if
your TCP layer is malformed.

Flow And Endpoint

gopacket has two useful objects, Flow and Endpoint, for communicating in a protocol
independent manner the fact that a packet is coming from A and going to B.
The general layer types LinkLayer, NetworkLayer, and TransportLayer all provide
methods for extracting their flow information, without worrying about the type
of the underlying Layer.

A Flow is a simple object made up of a set of two Endpoints, one source and one
destination.  It details the sender and receiver of the Layer of the Packet.

An Endpoint is a hashable representation of a source or destination.  For
example, for LayerTypeIPv4, an Endpoint contains the IP address bytes for a v4
IP packet.  A Flow can be broken into Endpoints, and Endpoints can be combined
into Flows:

 packet := gopacket.NewPacket(myPacketData, layers.LayerTypeEthernet, gopacket.Lazy)
 netFlow := packet.NetworkLayer().NetworkFlow()
 src, dst := netFlow.Endpoints()
 reverseFlow := gopacket.NewFlow(dst, src)

Both Endpoint and Flow objects can be used as map keys, and the equality
operator can compare them, so you can easily group together all packets
based on endpoint criteria:

 flows := map[gopacket.Endpoint]chan gopacket.Packet
 packet := gopacket.NewPacket(myPacketData, layers.LayerTypeEthernet, gopacket.Lazy)
 // Send all TCP packets to channels based on their destination port.
 if tcp := packet.Layer(layers.LayerTypeTCP); tcp != nil {
   flows[tcp.TransportFlow().Dst()] <- packet
 }
 // Look for all packets with the same source and destination network address
 if net := packet.NetworkLayer(); net != nil {
   src, dst := net.NetworkFlow().Endpoints()
   if src == dst {
     fmt.Println("Fishy packet has same network source and dst: %s", src)
   }
 }
 // Find all packets coming from UDP port 1000 to UDP port 500
 interestingFlow := gopacket.NewFlow(layers.NewUDPPortEndpoint(1000), layers.NewUDPPortEndpoint(500))
 if t := packet.NetworkLayer(); t != nil && t.TransportFlow() == interestingFlow {
   fmt.Println("Found that UDP flow I was looking for!")
 }

For load-balancing purposes, both Flow and Endpoint have FastHash() functions,
which provide quick, non-cryptographic hashes of their contents.  Of particular
importance is the fact that Flow FastHash() is symmetric: A->B will have the same
hash as B->A.  An example usage could be:

 channels := [8]chan gopacket.Packet
 for i := 0; i < 8; i++ {
   channels[i] = make(chan gopacket.Packet)
   go packetHandler(channels[i])
 }
 for packet := range getPackets() {
   if net := packet.NetworkLayer(); net != nil {
     channels[int(net.NetworkFlow().FastHash()) & 0x7] <- packet
   }
 }

This allows us to split up a packet stream while still making sure that each
stream sees all packets for a flow (and its bidirectional opposite).

Implementing Your Own Decoder

If your network has some strange encapsulation, you can implement your own
decoder.  In this example, we handle Ethernet packets which are encapsulated
in a 4-byte header.

 // Create a layer type, should be unique and high, so it doesn't conflict,
 // giving it a name and a decoder to use.
 var MyLayerType = gopacket.RegisterLayerType(12345, gopacket.LayerTypeMetadata{Name: "MyLayerType", Decoder: gopacket.DecodeFunc(decodeMyLayer)})

 // Implement my layer
 type MyLayer struct {
   StrangeHeader []byte
   payload []byte
 }
 func (m MyLayer) LayerType() gopacket.LayerType { return MyLayerType }
 func (m MyLayer) LayerContents() []byte { return m.StrangeHeader }
 func (m MyLayer) LayerPayload() []byte { return m.payload }

 // Now implement a decoder... this one strips off the first 4 bytes of the
 // packet.
 func decodeMyLayer(data []byte, p gopacket.PacketBuilder) error {
   // Create my layer
   p.AddLayer(&MyLayer{data[:4], data[4:]})
   // Determine how to handle the rest of the packet
   return p.NextDecoder(layers.LayerTypeEthernet)
 }

 // Finally, decode your packets:
 p := gopacket.NewPacket(data, MyLayerType, gopacket.Lazy)

See the docs for Decoder and PacketBuilder for more details on how coding
decoders works, or look at RegisterLayerType and RegisterEndpointType to see how
to add layer/endpoint types to gopacket.

Fast Decoding With DecodingLayerParser

TLDR:  DecodingLayerParser takes about 10% of the time as NewPacket to decode
packet data, but only for known packet stacks.

Basic decoding using gopacket.NewPacket or PacketSource.Packets is somewhat slow
due to its need to allocate a new packet and every respective layer.  It's very
versatile and can handle all known layer types, but sometimes you really only
care about a specific set of layers regardless, so that versatility is wasted.

DecodingLayerParser avoids memory allocation altogether by decoding packet
layers directly into preallocated objects, which you can then reference to get
the packet's information.  A quick example:

 func main() {
   var eth layers.Ethernet
   var ip4 layers.IPv4
   var ip6 layers.IPv6
   var tcp layers.TCP
   parser := gopacket.NewDecodingLayerParser(layers.LayerTypeEthernet, &eth, &ip4, &ip6, &tcp)
   decoded := []gopacket.LayerType{}
   for packetData := range somehowGetPacketData() {
     err := parser.DecodeLayers(packetData, &decoded)
     for _, layerType := range decoded {
       switch layerType {
         case layers.LayerTypeIPv6:
           fmt.Println("    IP6 ", ip6.SrcIP, ip6.DstIP)
         case layers.LayerTypeIPv4:
           fmt.Println("    IP4 ", ip4.SrcIP, ip4.DstIP)
       }
     }
   }
 }

The important thing to note here is that the parser is modifying the passed in
layers (eth, ip4, ip6, tcp) instead of allocating new ones, thus greatly
speeding up the decoding process.  It's even branching based on layer type...
it'll handle an (eth, ip4, tcp) or (eth, ip6, tcp) stack.  However, it won't
handle any other type... since no other decoders were passed in, an (eth, ip4,
udp) stack will stop decoding after ip4, and only pass back [LayerTypeEthernet,
LayerTypeIPv4] through the 'decoded' slice (along with an error saying it can't
decode a UDP packet).

Unfortunately, not all layers can be used by DecodingLayerParser... only those
implementing the DecodingLayer interface are usable.  Also, it's possible to
create DecodingLayers that are not themselves Layers... see
layers.IPv6ExtensionSkipper for an example of this.

Creating Packet Data

As well as offering the ability to decode packet data, gopacket will allow you
to create packets from scratch, as well.  A number of gopacket layers implement
the SerializableLayer interface; these layers can be serialized to a []byte in
the following manner:

  ip := &layers.IPv4{
    SrcIP: net.IP{1, 2, 3, 4},
    DstIP: net.IP{5, 6, 7, 8},
    // etc...
  }
  buf := gopacket.NewSerializeBuffer()
  opts := gopacket.SerializeOptions{}  // See SerializeOptions for more details.
  err := ip.SerializeTo(buf, opts)
  if err != nil { panic(err) }
  fmt.Println(buf.Bytes())  // prints out a byte slice containing the serialized IPv4 layer.

SerializeTo PREPENDS the given layer onto the SerializeBuffer, and they treat
the current buffer's Bytes() slice as the payload of the serializing layer.
Therefore, you can serialize an entire packet by serializing a set of layers in
reverse order (Payload, then TCP, then IP, then Ethernet, for example).  The
SerializeBuffer's SerializeLayers function is a helper that does exactly that.

To generate a (empty and useless, because no fields are set)
Ethernet(IPv4(TCP(Payload))) packet, for example, you can run:

  buf := gopacket.NewSerializeBuffer()
  opts := gopacket.SerializeOptions{}
  gopacket.SerializeLayers(buf, opts,
    &layers.Ethernet{},
    &layers.IPv4{},
    &layers.TCP{},
    gopacket.Payload([]byte{1, 2, 3, 4}))
  packetData := buf.Bytes()

A Final Note

If you use gopacket, you'll almost definitely want to make sure gopacket/layers
is imported, since when imported it sets all the LayerType variables and fills
in a lot of interesting variables/maps (DecodersByLayerName, etc).  Therefore,
it's recommended that even if you don't use any layers functions directly, you still import with:

  import (
    _ "github.com/google/gopacket/layers"
  )
*/
package gopacket
<commit_msg>handle error in documentation code<commit_after>// Copyright 2012 Google, Inc. All rights reserved.
//
// Use of this source code is governed by a BSD-style license
// that can be found in the LICENSE file in the root of the source
// tree.

/*
Package gopacket provides packet decoding for the Go language.

gopacket contains many sub-packages with additional functionality you may find
useful, including:

 * layers: You'll probably use this every time.  This contains of the logic
     built into gopacket for decoding packet protocols.  Note that all example
     code below assumes that you have imported both gopacket and
     gopacket/layers.
 * pcap: C bindings to use libpcap to read packets off the wire.
 * pfring: C bindings to use PF_RING to read packets off the wire.
 * afpacket: C bindings for Linux's AF_PACKET to read packets off the wire.
 * tcpassembly: TCP stream reassembly

Also, if you're looking to dive right into code, see the examples subdirectory
for numerous simple binaries built using gopacket libraries.

Basic Usage

gopacket takes in packet data as a []byte and decodes it into a packet with
a non-zero number of "layers".  Each layer corresponds to a protocol
within the bytes.  Once a packet has been decoded, the layers of the packet
can be requested from the packet.

 // Decode a packet
 packet := gopacket.NewPacket(myPacketData, layers.LayerTypeEthernet, gopacket.Default)
 // Get the TCP layer from this packet
 if tcpLayer := packet.Layer(layers.LayerTypeTCP); tcpLayer != nil {
   fmt.Println("This is a TCP packet!")
   // Get actual TCP data from this layer
   tcp, _ := tcpLayer.(*layers.TCP)
   fmt.Printf("From src port %d to dst port %d\n", tcp.SrcPort, tcp.DstPort)
 }
 // Iterate over all layers, printing out each layer type
 for _, layer := range packet.Layers() {
   fmt.Println("PACKET LAYER:", layer.LayerType())
 }

Packets can be decoded from a number of starting points.  Many of our base
types implement Decoder, which allow us to decode packets for which
we don't have full data.

 // Decode an ethernet packet
 ethP := gopacket.NewPacket(p1, layers.LayerTypeEthernet, gopacket.Default)
 // Decode an IPv6 header and everything it contains
 ipP := gopacket.NewPacket(p2, layers.LayerTypeIPv6, gopacket.Default)
 // Decode a TCP header and its payload
 tcpP := gopacket.NewPacket(p3, layers.LayerTypeTCP, gopacket.Default)

Reading Packets From A Source

Most of the time, you won't just have a []byte of packet data lying around.
Instead, you'll want to read packets in from somewhere (file, interface, etc)
and process them.  To do that, you'll want to build a PacketSource.

First, you'll need to construct an object that implements the PacketDataSource
interface.  There are implementations of this interface bundled with gopacket
in the gopacket/pcap and gopacket/pfring subpackages... see their documentation
for more information on their usage.  Once you have a PacketDataSource, you can
pass it into NewPacketSource, along with a Decoder of your choice, to create
a PacketSource.

Once you have a PacketSource, you can read packets from it in multiple ways.
See the docs for PacketSource for more details.  The easiest method is the
Packets function, which returns a channel, then asynchronously writes new
packets into that channel, closing the channel if the packetSource hits an
end-of-file.

  packetSource := ...  // construct using pcap or pfring
  for packet := range packetSource.Packets() {
    handlePacket(packet)  // do something with each packet
  }

You can change the decoding options of the packetSource by setting fields in
packetSource.DecodeOptions... see the following sections for more details.

Lazy Decoding

gopacket optionally decodes packet data lazily, meaning it
only decodes a packet layer when it needs to handle a function call.

 // Create a packet, but don't actually decode anything yet
 packet := gopacket.NewPacket(myPacketData, layers.LayerTypeEthernet, gopacket.Lazy)
 // Now, decode the packet up to the first IPv4 layer found but no further.
 // If no IPv4 layer was found, the whole packet will be decoded looking for
 // it.
 ip4 := packet.Layer(layers.LayerTypeIPv4)
 // Decode all layers and return them.  The layers up to the first IPv4 layer
 // are already decoded, and will not require decoding a second time.
 layers := packet.Layers()

Lazily-decoded packets are not concurrency-safe.  Since layers have not all been
decoded, each call to Layer() or Layers() has the potential to mutate the packet
in order to decode the next layer.  If a packet is used
in multiple goroutines concurrently, don't use gopacket.Lazy.  Then gopacket
will decode the packet fully, and all future function calls won't mutate the
object.

NoCopy Decoding

By default, gopacket will copy the slice passed to NewPacket and store the
copy within the packet, so future mutations to the bytes underlying the slice
don't affect the packet and its layers.  If you can guarantee that the
underlying slice bytes won't be changed, you can use NoCopy to tell
gopacket.NewPacket, and it'll use the passed-in slice itself.

 // This channel returns new byte slices, each of which points to a new
 // memory location that's guaranteed immutable for the duration of the
 // packet.
 for data := range myByteSliceChannel {
   p := gopacket.NewPacket(data, layers.LayerTypeEthernet, gopacket.NoCopy)
   doSomethingWithPacket(p)
 }

The fastest method of decoding is to use both Lazy and NoCopy, but note from
the many caveats above that for some implementations either or both may be
dangerous.

Pointers To Known Layers

During decoding, certain layers are stored in the packet as well-known
layer types.  For example, IPv4 and IPv6 are both considered NetworkLayer
layers, while TCP and UDP are both TransportLayer layers.  We support 4
layers, corresponding to the 4 layers of the TCP/IP layering scheme (roughly
anagalous to layers 2, 3, 4, and 7 of the OSI model).  To access these,
you can use the packet.LinkLayer, packet.NetworkLayer,
packet.TransportLayer, and packet.ApplicationLayer functions.  Each of
these functions returns a corresponding interface
(gopacket.{Link,Network,Transport,Application}Layer).  The first three
provide methods for getting src/dst addresses for that particular layer,
while the final layer provides a Payload function to get payload data.
This is helpful, for example, to get payloads for all packets regardless
of their underlying data type:

 // Get packets from some source
 for packet := range someSource {
   if app := packet.ApplicationLayer(); app != nil {
     if strings.Contains(string(app.Payload()), "magic string") {
       fmt.Println("Found magic string in a packet!")
     }
   }
 }

A particularly useful layer is ErrorLayer, which is set whenever there's
an error parsing part of the packet.

 packet := gopacket.NewPacket(myPacketData, layers.LayerTypeEthernet, gopacket.Default)
 if err := packet.ErrorLayer(); err != nil {
   fmt.Println("Error decoding some part of the packet:", err)
 }

Note that we don't return an error from NewPacket because we may have decoded
a number of layers successfully before running into our erroneous layer.  You
may still be able to get your Ethernet and IPv4 layers correctly, even if
your TCP layer is malformed.

Flow And Endpoint

gopacket has two useful objects, Flow and Endpoint, for communicating in a protocol
independent manner the fact that a packet is coming from A and going to B.
The general layer types LinkLayer, NetworkLayer, and TransportLayer all provide
methods for extracting their flow information, without worrying about the type
of the underlying Layer.

A Flow is a simple object made up of a set of two Endpoints, one source and one
destination.  It details the sender and receiver of the Layer of the Packet.

An Endpoint is a hashable representation of a source or destination.  For
example, for LayerTypeIPv4, an Endpoint contains the IP address bytes for a v4
IP packet.  A Flow can be broken into Endpoints, and Endpoints can be combined
into Flows:

 packet := gopacket.NewPacket(myPacketData, layers.LayerTypeEthernet, gopacket.Lazy)
 netFlow := packet.NetworkLayer().NetworkFlow()
 src, dst := netFlow.Endpoints()
 reverseFlow := gopacket.NewFlow(dst, src)

Both Endpoint and Flow objects can be used as map keys, and the equality
operator can compare them, so you can easily group together all packets
based on endpoint criteria:

 flows := map[gopacket.Endpoint]chan gopacket.Packet
 packet := gopacket.NewPacket(myPacketData, layers.LayerTypeEthernet, gopacket.Lazy)
 // Send all TCP packets to channels based on their destination port.
 if tcp := packet.Layer(layers.LayerTypeTCP); tcp != nil {
   flows[tcp.TransportFlow().Dst()] <- packet
 }
 // Look for all packets with the same source and destination network address
 if net := packet.NetworkLayer(); net != nil {
   src, dst := net.NetworkFlow().Endpoints()
   if src == dst {
     fmt.Println("Fishy packet has same network source and dst: %s", src)
   }
 }
 // Find all packets coming from UDP port 1000 to UDP port 500
 interestingFlow := gopacket.NewFlow(layers.NewUDPPortEndpoint(1000), layers.NewUDPPortEndpoint(500))
 if t := packet.NetworkLayer(); t != nil && t.TransportFlow() == interestingFlow {
   fmt.Println("Found that UDP flow I was looking for!")
 }

For load-balancing purposes, both Flow and Endpoint have FastHash() functions,
which provide quick, non-cryptographic hashes of their contents.  Of particular
importance is the fact that Flow FastHash() is symmetric: A->B will have the same
hash as B->A.  An example usage could be:

 channels := [8]chan gopacket.Packet
 for i := 0; i < 8; i++ {
   channels[i] = make(chan gopacket.Packet)
   go packetHandler(channels[i])
 }
 for packet := range getPackets() {
   if net := packet.NetworkLayer(); net != nil {
     channels[int(net.NetworkFlow().FastHash()) & 0x7] <- packet
   }
 }

This allows us to split up a packet stream while still making sure that each
stream sees all packets for a flow (and its bidirectional opposite).

Implementing Your Own Decoder

If your network has some strange encapsulation, you can implement your own
decoder.  In this example, we handle Ethernet packets which are encapsulated
in a 4-byte header.

 // Create a layer type, should be unique and high, so it doesn't conflict,
 // giving it a name and a decoder to use.
 var MyLayerType = gopacket.RegisterLayerType(12345, gopacket.LayerTypeMetadata{Name: "MyLayerType", Decoder: gopacket.DecodeFunc(decodeMyLayer)})

 // Implement my layer
 type MyLayer struct {
   StrangeHeader []byte
   payload []byte
 }
 func (m MyLayer) LayerType() gopacket.LayerType { return MyLayerType }
 func (m MyLayer) LayerContents() []byte { return m.StrangeHeader }
 func (m MyLayer) LayerPayload() []byte { return m.payload }

 // Now implement a decoder... this one strips off the first 4 bytes of the
 // packet.
 func decodeMyLayer(data []byte, p gopacket.PacketBuilder) error {
   // Create my layer
   p.AddLayer(&MyLayer{data[:4], data[4:]})
   // Determine how to handle the rest of the packet
   return p.NextDecoder(layers.LayerTypeEthernet)
 }

 // Finally, decode your packets:
 p := gopacket.NewPacket(data, MyLayerType, gopacket.Lazy)

See the docs for Decoder and PacketBuilder for more details on how coding
decoders works, or look at RegisterLayerType and RegisterEndpointType to see how
to add layer/endpoint types to gopacket.

Fast Decoding With DecodingLayerParser

TLDR:  DecodingLayerParser takes about 10% of the time as NewPacket to decode
packet data, but only for known packet stacks.

Basic decoding using gopacket.NewPacket or PacketSource.Packets is somewhat slow
due to its need to allocate a new packet and every respective layer.  It's very
versatile and can handle all known layer types, but sometimes you really only
care about a specific set of layers regardless, so that versatility is wasted.

DecodingLayerParser avoids memory allocation altogether by decoding packet
layers directly into preallocated objects, which you can then reference to get
the packet's information.  A quick example:

 func main() {
   var eth layers.Ethernet
   var ip4 layers.IPv4
   var ip6 layers.IPv6
   var tcp layers.TCP
   parser := gopacket.NewDecodingLayerParser(layers.LayerTypeEthernet, &eth, &ip4, &ip6, &tcp)
   decoded := []gopacket.LayerType{}
   for packetData := range somehowGetPacketData() {
     if err := parser.DecodeLayers(packetData, &decoded); err != nil {
       fmt.Fprintf(os.Stderr, "Could not decode layers: %v\n", err)
       continue
     }
     for _, layerType := range decoded {
       switch layerType {
         case layers.LayerTypeIPv6:
           fmt.Println("    IP6 ", ip6.SrcIP, ip6.DstIP)
         case layers.LayerTypeIPv4:
           fmt.Println("    IP4 ", ip4.SrcIP, ip4.DstIP)
       }
     }
   }
 }

The important thing to note here is that the parser is modifying the passed in
layers (eth, ip4, ip6, tcp) instead of allocating new ones, thus greatly
speeding up the decoding process.  It's even branching based on layer type...
it'll handle an (eth, ip4, tcp) or (eth, ip6, tcp) stack.  However, it won't
handle any other type... since no other decoders were passed in, an (eth, ip4,
udp) stack will stop decoding after ip4, and only pass back [LayerTypeEthernet,
LayerTypeIPv4] through the 'decoded' slice (along with an error saying it can't
decode a UDP packet).

Unfortunately, not all layers can be used by DecodingLayerParser... only those
implementing the DecodingLayer interface are usable.  Also, it's possible to
create DecodingLayers that are not themselves Layers... see
layers.IPv6ExtensionSkipper for an example of this.

Creating Packet Data

As well as offering the ability to decode packet data, gopacket will allow you
to create packets from scratch, as well.  A number of gopacket layers implement
the SerializableLayer interface; these layers can be serialized to a []byte in
the following manner:

  ip := &layers.IPv4{
    SrcIP: net.IP{1, 2, 3, 4},
    DstIP: net.IP{5, 6, 7, 8},
    // etc...
  }
  buf := gopacket.NewSerializeBuffer()
  opts := gopacket.SerializeOptions{}  // See SerializeOptions for more details.
  err := ip.SerializeTo(buf, opts)
  if err != nil { panic(err) }
  fmt.Println(buf.Bytes())  // prints out a byte slice containing the serialized IPv4 layer.

SerializeTo PREPENDS the given layer onto the SerializeBuffer, and they treat
the current buffer's Bytes() slice as the payload of the serializing layer.
Therefore, you can serialize an entire packet by serializing a set of layers in
reverse order (Payload, then TCP, then IP, then Ethernet, for example).  The
SerializeBuffer's SerializeLayers function is a helper that does exactly that.

To generate a (empty and useless, because no fields are set)
Ethernet(IPv4(TCP(Payload))) packet, for example, you can run:

  buf := gopacket.NewSerializeBuffer()
  opts := gopacket.SerializeOptions{}
  gopacket.SerializeLayers(buf, opts,
    &layers.Ethernet{},
    &layers.IPv4{},
    &layers.TCP{},
    gopacket.Payload([]byte{1, 2, 3, 4}))
  packetData := buf.Bytes()

A Final Note

If you use gopacket, you'll almost definitely want to make sure gopacket/layers
is imported, since when imported it sets all the LayerType variables and fills
in a lot of interesting variables/maps (DecodersByLayerName, etc).  Therefore,
it's recommended that even if you don't use any layers functions directly, you still import with:

  import (
    _ "github.com/google/gopacket/layers"
  )
*/
package gopacket
<|endoftext|>


<commit_before>// Copyright 2017 Tomas Machalek <tomas.machalek@gmail.com>
// Copyright 2017 Charles University, Faculty of Arts,
//                Institute of the Czech National Corpus
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
	"encoding/json"
	"flag"
	"fmt"
	"log"
	"os"
	"strings"
	"time"

	"github.com/czcorpus/vert-tagextract/db"
	"github.com/czcorpus/vert-tagextract/db/colgen"
	"github.com/czcorpus/vert-tagextract/vteconf"
	"github.com/tomachalek/vertigo"
)

func dumpNewConf(dstPath string) {
	conf := vteconf.VTEConf{}
	conf.Encoding = "UTF-8"
	conf.AtomStructure = "p"
	conf.Structures = make(map[string][]string)
	conf.Structures["doc"] = []string{"id", "title"}
	conf.Structures["p"] = []string{"id", "type"}
	conf.IndexedCols = []string{}
	conf.BibView.Cols = []string{"doc_id", "doc_title", "doc_author", "doc_publisher"}
	conf.BibView.IDAttr = "doc_id"
	conf.SelfJoin.ArgColumns = []string{}
	b, err := json.MarshalIndent(conf, "", "  ")
	if err != nil {
		log.Fatalf("Failed to dump a new config: %s", err)
	}
	f, err := os.OpenFile(dstPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0664)
	if err != nil {
		log.Fatalf("Failed to open file %s for writing", dstPath)
	}
	defer f.Close()
	f.Write(b)
}

func exportData(confPath string, appendData bool) {
	conf := vteconf.LoadConf(confPath)

	_, ferr := os.Stat(conf.DBFile)
	if os.IsNotExist(ferr) {
		if appendData {
			log.Fatalf("Update flag is set but the database %s does not exist", conf.DBFile)
		}

	} else if !appendData {
		log.Printf("The database file %s already exists. Existing data will be deleted.", conf.DBFile)
	}
	dbConn := db.OpenDatabase(conf.DBFile)

	if !appendData {
		db.DropExisting(dbConn)
		db.CreateSchema(dbConn, conf)
		if conf.HasConfiguredBib() {
			db.CreateBibView(dbConn, conf)
		}
	}

	parserConf := &vertigo.ParserConf{
		VerticalFilePath:      conf.VerticalFile,
		StructAttrAccumulator: "nil",
		Encoding:              conf.Encoding,
	}

	var fn colgen.AlignedColGenFn
	if conf.UsesSelfJoin() {
		fn = func(args map[string]interface{}) string {
			return colgen.GetFuncByName(conf.SelfJoin.GeneratorFn)(args, conf.SelfJoin.ArgColumns)
		}
	}

	tte := db.NewTTExtractor(dbConn, conf.Corpus, conf.AtomStructure, conf.StackStructEval, conf.Structures, fn)
	t0 := time.Now()
	tte.Run(parserConf)
	log.Printf("Finished in %s.\n", time.Since(t0))
}

func main() {
	flag.Usage = func() {
		fmt.Println("\n+-------------------------------------------------------------+")
		fmt.Println("| Vert-tagextract (vte) - a program for extracting structural |")
		fmt.Println("|            meta-data from a corpus vertical file            |")
		fmt.Println("|                         version 0.2                         |")
		fmt.Println("|          (c) Institute of the Czech National Corpus         |")
		fmt.Println("|         (c) Tomas Machalek tomas.machalek@ff.cuni.cz        |")
		fmt.Println("+-------------------------------------------------------------+")
		fmt.Printf("\nSupported encodings:\n%s\n", strings.Join(vertigo.SupportedCharsets(), ", "))
		fmt.Printf("\nSupported selfJoin column generator functions:\n%s\n", strings.Join(colgen.GetFuncList(), ", "))
		fmt.Println("\nUsage:")
		fmt.Println("vte create config.json\n\t(run an export configured in config.json, add data to a new database)")
		fmt.Println("vte append config.json\n\t(run an export configured in config.json, add data to an existing database)")
		fmt.Println("vte template config.json\n\t(create a half empty sample config config.json)")
		fmt.Println("\n(config file should be named after a respective corpus name, e.g. syn_v4.json)")

		fmt.Println("\nOptions:")
		flag.PrintDefaults()
	}

	flag.Parse()
	if len(flag.Args()) == 2 {
		switch flag.Arg(0) {
		case "create":
			exportData(flag.Arg(1), false)
		case "append":
			exportData(flag.Arg(1), true)
		case "template":
			dumpNewConf(flag.Arg(1))
		default:
			log.Fatalf("Unknown command '%s'", flag.Arg(0))
		}

	} else {
		log.Fatal("Unknown arguments, an action and a config file must be specified (use -h for help)")
	}

}
<commit_msg>Fixed property<commit_after>// Copyright 2017 Tomas Machalek <tomas.machalek@gmail.com>
// Copyright 2017 Charles University, Faculty of Arts,
//                Institute of the Czech National Corpus
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
	"encoding/json"
	"flag"
	"fmt"
	"log"
	"os"
	"strings"
	"time"

	"github.com/czcorpus/vert-tagextract/db"
	"github.com/czcorpus/vert-tagextract/db/colgen"
	"github.com/czcorpus/vert-tagextract/vteconf"
	"github.com/tomachalek/vertigo"
)

func dumpNewConf(dstPath string) {
	conf := vteconf.VTEConf{}
	conf.Encoding = "UTF-8"
	conf.AtomStructure = "p"
	conf.Structures = make(map[string][]string)
	conf.Structures["doc"] = []string{"id", "title"}
	conf.Structures["p"] = []string{"id", "type"}
	conf.IndexedCols = []string{}
	conf.BibView.Cols = []string{"doc_id", "doc_title", "doc_author", "doc_publisher"}
	conf.BibView.IDAttr = "doc_id"
	conf.SelfJoin.ArgColumns = []string{}
	b, err := json.MarshalIndent(conf, "", "  ")
	if err != nil {
		log.Fatalf("Failed to dump a new config: %s", err)
	}
	f, err := os.OpenFile(dstPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0664)
	if err != nil {
		log.Fatalf("Failed to open file %s for writing", dstPath)
	}
	defer f.Close()
	f.Write(b)
}

func exportData(confPath string, appendData bool) {
	conf := vteconf.LoadConf(confPath)

	_, ferr := os.Stat(conf.DBFile)
	if os.IsNotExist(ferr) {
		if appendData {
			log.Fatalf("Update flag is set but the database %s does not exist", conf.DBFile)
		}

	} else if !appendData {
		log.Printf("The database file %s already exists. Existing data will be deleted.", conf.DBFile)
	}
	dbConn := db.OpenDatabase(conf.DBFile)

	if !appendData {
		db.DropExisting(dbConn)
		db.CreateSchema(dbConn, conf)
		if conf.HasConfiguredBib() {
			db.CreateBibView(dbConn, conf)
		}
	}

	parserConf := &vertigo.ParserConf{
		InputFilePath:         conf.VerticalFile,
		StructAttrAccumulator: "nil",
		Encoding:              conf.Encoding,
	}

	var fn colgen.AlignedColGenFn
	if conf.UsesSelfJoin() {
		fn = func(args map[string]interface{}) string {
			return colgen.GetFuncByName(conf.SelfJoin.GeneratorFn)(args, conf.SelfJoin.ArgColumns)
		}
	}

	tte := db.NewTTExtractor(dbConn, conf.Corpus, conf.AtomStructure, conf.StackStructEval, conf.Structures, fn)
	t0 := time.Now()
	tte.Run(parserConf)
	log.Printf("Finished in %s.\n", time.Since(t0))
}

func main() {
	flag.Usage = func() {
		fmt.Println("\n+-------------------------------------------------------------+")
		fmt.Println("| Vert-tagextract (vte) - a program for extracting structural |")
		fmt.Println("|            meta-data from a corpus vertical file            |")
		fmt.Println("|                         version 0.2                         |")
		fmt.Println("|          (c) Institute of the Czech National Corpus         |")
		fmt.Println("|         (c) Tomas Machalek tomas.machalek@ff.cuni.cz        |")
		fmt.Println("+-------------------------------------------------------------+")
		fmt.Printf("\nSupported encodings:\n%s\n", strings.Join(vertigo.SupportedCharsets(), ", "))
		fmt.Printf("\nSupported selfJoin column generator functions:\n%s\n", strings.Join(colgen.GetFuncList(), ", "))
		fmt.Println("\nUsage:")
		fmt.Println("vte create config.json\n\t(run an export configured in config.json, add data to a new database)")
		fmt.Println("vte append config.json\n\t(run an export configured in config.json, add data to an existing database)")
		fmt.Println("vte template config.json\n\t(create a half empty sample config config.json)")
		fmt.Println("\n(config file should be named after a respective corpus name, e.g. syn_v4.json)")

		fmt.Println("\nOptions:")
		flag.PrintDefaults()
	}

	flag.Parse()
	if len(flag.Args()) == 2 {
		switch flag.Arg(0) {
		case "create":
			exportData(flag.Arg(1), false)
		case "append":
			exportData(flag.Arg(1), true)
		case "template":
			dumpNewConf(flag.Arg(1))
		default:
			log.Fatalf("Unknown command '%s'", flag.Arg(0))
		}

	} else {
		log.Fatal("Unknown arguments, an action and a config file must be specified (use -h for help)")
	}

}
<|endoftext|>


<commit_before>package dns

import (
	"time"
)

// TODO: axfrreceive fixen, first can go
// a tsigTimersonly to responsewriter

// XfrMsg is used when doing [IA]xfr with a remote server.
type XfrMsg struct {
	RR    []RR          // the set of RRs in the answer section form the message of the server
	Rtt   time.Duration // round trip time
	Error error         // if something went wrong, this contains the error
}

// XfrReceive performs a [AI]xfr request (depends on the message's Qtype). It returns
// a channel of XfrMsg on which the replies from the server are sent. At the end of
// the transfer the channel is closed.
// It panics if the Qtype does not equal TypeAXFR or TypeIXFR. The messages are TSIG checked if
// needed, no other post-processing is performed. The caller must dissect the returned
// messages.
//
// Basic use pattern for receiving an AXFR:
//
//	// m contains the AXFR request
//	t, e := client.XfrReceive(m, "127.0.0.1:53")
//	for r := range t {
//		// ... deal with r.RR or r.Error
//	}
func (c *Client) XfrReceive(q *Msg, a string) (chan *XfrMsg, error) {
	w := new(reply)
	w.client = c
	w.addr = a
	w.req = q
	if err := w.dial(); err != nil {
		return nil, err
	}
	if err := w.send(q); err != nil {
		return nil, err
	}
	e := make(chan *XfrMsg)
	switch q.Question[0].Qtype {
	case TypeAXFR:
		go w.axfrReceive(e)
		return e, nil
	case TypeIXFR:
		go w.ixfrReceive(e)
		return e, nil
	default:
		return nil, ErrXfrType
	}
	panic("not reached")
}

func (w *reply) axfrReceive(c chan *XfrMsg) {
	first := true
	defer w.conn.Close()
	defer close(c)
	for {
		in, err := w.receive()
		if err != nil {
			c <- &XfrMsg{RR: in.Answer, Rtt: w.rtt, Error: err}
			return
		}
		if w.req.Id != in.Id {
			c <- &XfrMsg{RR: in.Answer, Rtt: w.rtt, Error: ErrId}
			return
		}
		if first {
			if !checkXfrSOA(in, true) {
				c <- &XfrMsg{RR: in.Answer, Rtt: w.rtt, Error: ErrXfrSoa}
				return
			}
			first = !first
		}

		if !first {
			w.tsigTimersOnly = true // Subsequent envelopes use this.
			if checkXfrSOA(in, false) {
				c <- &XfrMsg{RR: in.Answer, Rtt: w.rtt, Error: nil}
				return
			}
			c <- &XfrMsg{RR: in.Answer, Rtt: w.rtt, Error: nil}
		}
	}
	panic("not reached")
}

func (w *reply) ixfrReceive(c chan *XfrMsg) {
	var serial uint32 // The first serial seen is the current server serial
	first := true
	defer w.conn.Close()
	defer close(c)
	for {
		in, err := w.receive()
		if err != nil {
			c <- &XfrMsg{RR: in.Answer, Rtt: w.rtt, Error: err}
			return
		}
		if w.req.Id != in.Id {
			c <- &XfrMsg{RR: in.Answer, Rtt: w.rtt, Error: ErrId}
			return
		}
		if first {
			// A single SOA RR signals "no changes"
			if len(in.Answer) == 1 && checkXfrSOA(in, true) {
				c <- &XfrMsg{RR: in.Answer, Rtt: w.rtt, Error: nil}
				return
			}

			// Check if the returned answer is ok
			if !checkXfrSOA(in, true) {
				c <- &XfrMsg{RR: in.Answer, Rtt: w.rtt, Error: ErrXfrSoa}
				return
			}
			// This serial is important
			serial = in.Answer[0].(*RR_SOA).Serial
			first = !first
		}

		// Now we need to check each message for SOA records, to see what we need to do
		if !first {
			w.tsigTimersOnly = true
			// If the last record in the IXFR contains the servers' SOA,  we should quit
			if v, ok := in.Answer[len(in.Answer)-1].(*RR_SOA); ok {
				if v.Serial == serial {
					c <- &XfrMsg{RR: in.Answer, Rtt: w.rtt, Error: nil}
					return
				}
			}
			c <- &XfrMsg{RR: in.Answer, Rtt: w.rtt}
		}
	}
	panic("not reached")
}

// XfrSend performs an outgoing [IX]xfr depending on the request message. As
// long as the channel c is open ... TODO(mg): docs
// tsig is done, enveloping is done, voor de rest niks... TODO
func XfrSend(w ResponseWriter, req *Msg, c chan *XfrMsg) error {
	switch req.Question[0].Qtype {
	case TypeAXFR, TypeIXFR:
		go axfrSend(w, req, c)
	default:
		return ErrXfrType
	}
	return nil
}

// TODO(mg): count the RRs and the resulting size.
func axfrSend(w ResponseWriter, req *Msg, c chan *XfrMsg) {
	rep := new(Msg)
	rep.SetReply(req)
	rep.MsgHdr.Authoritative = true

	first := true
	w.TsigTimersOnly(false)
	for x := range c {
		// assume is fits
		rep.Answer = append(rep.Answer, x.RR...)
		w.Write(rep)
		if first {
			first = !first
			w.TsigTimersOnly(first)
		}
		rep.Answer = nil
	}
}

// Check if he SOA record exists in the Answer section of
// the packet. If first is true the first RR must be a SOA
// if false, the last one should be a SOA.
func checkXfrSOA(in *Msg, first bool) bool {
	if len(in.Answer) > 0 {
		if first {
			return in.Answer[0].Header().Rrtype == TypeSOA
		} else {
			return in.Answer[len(in.Answer)-1].Header().Rrtype == TypeSOA
		}
	}
	return false
}
<commit_msg>rename XfrMsg to XfrToken<commit_after>package dns

// XfrToken is used when doing [IA]xfr with a remote server.
type XfrToken struct {
	RR    []RR          // the set of RRs in the answer section form the message of the server
	Error error         // if something went wrong, this contains the error
}

// XfrReceive performs a [AI]xfr request (depends on the message's Qtype). It returns
// a channel of XfrToken on which the replies from the server are sent. At the end of
// the transfer the channel is closed.
// It panics if the Qtype does not equal TypeAXFR or TypeIXFR. The messages are TSIG checked if
// needed, no other post-processing is performed. The caller must dissect the returned
// messages.
//
// Basic use pattern for receiving an AXFR:
//
//	// m contains the AXFR request
//	t, e := client.XfrReceive(m, "127.0.0.1:53")
//	for r := range t {
//		// ... deal with r.RR or r.Error
//	}
func (c *Client) XfrReceive(q *Msg, a string) (chan *XfrToken, error) {
	w := new(reply)
	w.client = c
	w.addr = a
	w.req = q
	if err := w.dial(); err != nil {
		return nil, err
	}
	if err := w.send(q); err != nil {
		return nil, err
	}
	e := make(chan *XfrToken)
	switch q.Question[0].Qtype {
	case TypeAXFR:
		go w.axfrReceive(e)
		return e, nil
	case TypeIXFR:
		go w.ixfrReceive(e)
		return e, nil
	default:
		return nil, ErrXfrType
	}
	panic("not reached")
}

func (w *reply) axfrReceive(c chan *XfrToken) {
	first := true
	defer w.conn.Close()
	defer close(c)
	for {
		in, err := w.receive()
		if err != nil {
			c <- &XfrToken{in.Answer, err}
			return
		}
		if w.req.Id != in.Id {
			c <- &XfrToken{in.Answer, ErrId}
			return
		}
		if first {
			if !checkXfrSOA(in, true) {
				c <- &XfrToken{in.Answer, ErrXfrSoa}
				return
			}
			first = !first
		}

		if !first {
			w.tsigTimersOnly = true // Subsequent envelopes use this.
			if checkXfrSOA(in, false) {
				c <- &XfrToken{in.Answer, nil}
				return
			}
			c <- &XfrToken{in.Answer, nil}
		}
	}
	panic("not reached")
}

func (w *reply) ixfrReceive(c chan *XfrToken) {
	var serial uint32 // The first serial seen is the current server serial
	first := true
	defer w.conn.Close()
	defer close(c)
	for {
		in, err := w.receive()
		if err != nil {
			c <- &XfrToken{in.Answer, err}
			return
		}
		if w.req.Id != in.Id {
			c <- &XfrToken{in.Answer, ErrId}
			return
		}
		if first {
			// A single SOA RR signals "no changes"
			if len(in.Answer) == 1 && checkXfrSOA(in, true) {
				c <- &XfrToken{in.Answer, nil}
				return
			}

			// Check if the returned answer is ok
			if !checkXfrSOA(in, true) {
				c <- &XfrToken{in.Answer, ErrXfrSoa}
				return
			}
			// This serial is important
			serial = in.Answer[0].(*RR_SOA).Serial
			first = !first
		}

		// Now we need to check each message for SOA records, to see what we need to do
		if !first {
			w.tsigTimersOnly = true
			// If the last record in the IXFR contains the servers' SOA,  we should quit
			if v, ok := in.Answer[len(in.Answer)-1].(*RR_SOA); ok {
				if v.Serial == serial {
					c <- &XfrToken{in.Answer, nil}
					return
				}
			}
			c <- &XfrToken{in.Answer, nil}
		}
	}
	panic("not reached")
}

// Check if he SOA record exists in the Answer section of
// the packet. If first is true the first RR must be a SOA
// if false, the last one should be a SOA.
func checkXfrSOA(in *Msg, first bool) bool {
	if len(in.Answer) > 0 {
		if first {
			return in.Answer[0].Header().Rrtype == TypeSOA
		} else {
			return in.Answer[len(in.Answer)-1].Header().Rrtype == TypeSOA
		}
	}
	return false
}

// XfrSend performs an outgoing [IX]xfr depending on the request message. As
// long as the channel c is open ... TODO(mg): docs
// tsig is done, enveloping is done, voor de rest niks... TODO
func XfrSend(w ResponseWriter, req *Msg, c chan *XfrToken) error {
	switch req.Question[0].Qtype {
	case TypeAXFR, TypeIXFR:
		go axfrSend(w, req, c)
	default:
		return ErrXfrType
	}
	return nil
}

// TODO(mg): count the RRs and the resulting size.
func axfrSend(w ResponseWriter, req *Msg, c chan *XfrToken) {
	rep := new(Msg)
	rep.SetReply(req)
	rep.MsgHdr.Authoritative = true

	first := true
	w.TsigTimersOnly(false)
	for x := range c {
		// assume it fits
		rep.Answer = append(rep.Answer, x.RR...)
		w.Write(rep)
		if first {
			first = !first
			w.TsigTimersOnly(first)
		}
		rep.Answer = nil
	}
}

<|endoftext|>


<commit_before>package xpp

import (
	"encoding/xml"
	"errors"
	"fmt"
	"io"
	"strings"
)

type XMLEventType int

const (
	StartDocument XMLEventType = iota
	EndDocument
	StartTag
	EndTag
	Text
	Comment
	ProcessingInstruction
	Directive
	IgnorableWhitespace // TODO: ?
	// TODO: CDSECT ?
)

type XMLPullParser struct {
	Decoder *xml.Decoder

	// Document State
	Spaces map[string]string

	// Token State
	Depth int
	Event XMLEventType
	Attrs []xml.Attr
	Name  string
	Space string
	Text  string

	token     interface{}
	peekToken interface{}
	peekEvent XMLEventType
	peekErr   error
}

func NewXMLPullParser(r io.Reader) *XMLPullParser {
	d := xml.NewDecoder(r)
	return &XMLPullParser{
		Decoder: d,
		Event:   StartDocument,
		Depth:   0,
		Spaces:  map[string]string{},
	}
}

func (p *XMLPullParser) NextTag() (event XMLEventType, err error) {
	t, err := p.Next()
	if err != nil {
		return event, err
	}

	if t == Text && p.isWhitespace() {
		t, err = p.Next()
		if err != nil {
			return event, err
		}
	}

	if t != StartTag && t != EndTag {
		return event, fmt.Errorf("Expected StartTag or EndTag but got %s", p.eventName(t))
	}

	return t, nil
}

func (p *XMLPullParser) Next() (event XMLEventType, err error) {
	text := ""

	for {
		event, err = p.NextToken()
		if err != nil {
			return event, err
		}

		// Return immediately after encountering a StartTag
		// EndTag or EndDocument
		if event == StartTag ||
			event == EndTag ||
			event == EndDocument {
			return event, nil
		}

		if event == Comment ||
			event == Directive ||
			event == ProcessingInstruction {
			text = ""
			continue
		}

		// Coalesce text event value
		if event == Text {
			text += p.Text
		}

		// Return the text event if it is going to be
		// ended by a Start/EndTag or EndDocument.
		// Otherwise we will continue to coalesce text
		// events.
		if text != "" && (p.peekEvent == StartTag ||
			p.peekEvent == EndTag ||
			p.peekEvent == EndDocument) {
			p.Text = text
			return Text, nil
		}
	}
	return event, nil
}

func (p *XMLPullParser) NextToken() (event XMLEventType, err error) {
	// Clear any state held for the previous token
	p.resetTokenState()

	if p.peekErr != nil {
		return event, p.peekErr
	}

	if p.Event == StartDocument {
		// Preload the peek token when first starting
		p.peekNextToken()
	} else if p.Event == EndDocument {
		// Always return EndDocument once that state has
		// been reached.
		return p.Event, nil
	}

	// Switch peek token/event to the current token/event
	p.Event = p.peekEvent
	p.token = p.peekToken
	p.processToken(p.token)

	p.peekNextToken()

	// Return current event (previously the peek token)
	return p.Event, nil
}

func (p *XMLPullParser) peekNextToken() {
	// Peek the next token/event
	peekToken, err := p.Decoder.Token()
	if err != nil {
		if err == io.EOF {
			// XML decoder returns the EOF as an error
			// but we want to return it as a valid
			// EndDocument token instead
			p.peekToken = nil
			p.peekEvent = EndDocument
		} else {
			p.peekErr = err
		}
		return
	}
	p.peekToken = xml.CopyToken(peekToken)
	p.peekEvent = p.eventType(peekToken)
}

func (p *XMLPullParser) NextText() (string, error) {
	if p.Event != StartTag {
		return "", errors.New("Parser must be on StartTag to get NextText()")
	}

	t, err := p.Next()
	if err != nil {
		return "", err
	}

	if t == Text {
		result := p.Text
		nt, err := p.Next()
		if err != nil {
			return "", err
		}

		if nt != EndTag {
			return "", errors.New("Event Text must be immediately followed by EndTag")
		}

		return result, nil
	} else if t == EndTag {
		return "", nil
	} else {
		return "", errors.New("Parser must be on StartTag or Text to read text")
	}
}

func (p *XMLPullParser) Skip() error {
	for {
		tok, err := p.NextToken()
		if err != nil {
			return err
		}
		if tok == StartTag {
			if err := p.Skip(); err != nil {
				return err
			}
		} else if tok == EndTag {
			return nil
		}
	}
}

func (p *XMLPullParser) Attribute(name string) string {
	for _, attr := range p.Attrs {
		if attr.Name.Local == name {
			return attr.Value
		}
	}
	return ""
}

func (p *XMLPullParser) Expect(event XMLEventType, name string) (err error) {
	return p.ExpectAll(event, "*", name)
}

func (p *XMLPullParser) ExpectAll(event XMLEventType, space string, name string) (err error) {
	if !(p.Event == event && (p.Space == space || space == "*") && (p.Name == name || name == "*")) {
		err = fmt.Errorf("Expected Space:%s Name:%s Event:%s but got Space:%s Name:%s Event:%s", space, name, p.eventName(event), p.Space, p.Name, p.eventName(p.Event))
	}
	return
}

func (p *XMLPullParser) processToken(t xml.Token) {
	switch tt := t.(type) {
	case xml.StartElement:
		p.processStartToken(tt)
	case xml.EndElement:
		p.processEndToken(tt)
	case xml.CharData:
		p.processCharDataToken(tt)
	case xml.Comment:
		p.processCommentToken(tt)
	case xml.ProcInst:
		p.processProcInstToken(tt)
	case xml.Directive:
		p.processDirectiveToken(tt)
	}
}

func (p *XMLPullParser) processStartToken(t xml.StartElement) {
	p.Depth++
	p.Attrs = t.Attr
	p.Name = t.Name.Local
	p.Space = t.Name.Space
	p.trackNamespaces(t)
}

func (p *XMLPullParser) processEndToken(t xml.EndElement) {
	p.Depth--
	p.Name = t.Name.Local
}

func (p *XMLPullParser) processCharDataToken(t xml.CharData) {
	p.Text = string([]byte(t))
}

func (p *XMLPullParser) processCommentToken(t xml.Comment) {
	p.Text = string([]byte(t))
}

func (p *XMLPullParser) processProcInstToken(t xml.ProcInst) {
	p.Text = fmt.Sprintf("%s %s", t.Target, string(t.Inst))
}

func (p *XMLPullParser) processDirectiveToken(t xml.Directive) {
	p.Text = string([]byte(t))
}

func (p *XMLPullParser) resetTokenState() {
	p.Attrs = nil
	p.Name = ""
	p.Space = ""
	p.Text = ""
}

func (p *XMLPullParser) isWhitespace() bool {
	return strings.TrimSpace(p.Text) == ""
}

func (p *XMLPullParser) eventName(e XMLEventType) (name string) {
	switch e {
	case StartTag:
		name = "StartTag"
	case EndTag:
		name = "EndTag"
	case StartDocument:
		name = "StartDocument"
	case EndDocument:
		name = "EndDocument"
	case ProcessingInstruction:
		name = "ProcessingInstruction"
	case Directive:
		name = "Directive"
	case Comment:
		name = "Comment"
	case Text:
		name = "Text"
	case IgnorableWhitespace:
		name = "IgnorableWhitespace"
	}
	return
}

func (p *XMLPullParser) eventType(t xml.Token) (event XMLEventType) {
	switch t.(type) {
	case xml.StartElement:
		event = StartTag
	case xml.EndElement:
		event = EndTag
	case xml.CharData:
		event = Text
	case xml.Comment:
		event = Comment
	case xml.ProcInst:
		event = ProcessingInstruction
	case xml.Directive:
		event = Directive
	}
	return
}

func (p *XMLPullParser) trackNamespaces(t xml.StartElement) {
	for _, attr := range t.Attr {
		if attr.Name.Space == "xmlns" {
			space := strings.TrimSpace(attr.Value)
			spacePrefix := strings.TrimSpace(strings.ToLower(attr.Name.Local))
			p.Spaces[space] = spacePrefix
		} else if attr.Name.Local == "xmlns" {
			space := strings.TrimSpace(attr.Value)
			p.Spaces[space] = ""
		}
	}
}
<commit_msg>Decode element<commit_after>package xpp

import (
	"encoding/xml"
	"errors"
	"fmt"
	"io"
	"strings"
)

type XMLEventType int

const (
	StartDocument XMLEventType = iota
	EndDocument
	StartTag
	EndTag
	Text
	Comment
	ProcessingInstruction
	Directive
	IgnorableWhitespace // TODO: ?
	// TODO: CDSECT ?
)

type XMLPullParser struct {
	// Document State
	Spaces map[string]string

	// Token State
	Depth int
	Event XMLEventType
	Attrs []xml.Attr
	Name  string
	Space string
	Text  string

	decoder   *xml.Decoder
	token     interface{}
	peekToken interface{}
	peekEvent XMLEventType
	peekErr   error
}

func NewXMLPullParser(r io.Reader, strict bool) *XMLPullParser {
	d := xml.NewDecoder(r)
	d.Strict = strict
	return &XMLPullParser{
		decoder: d,
		Event:   StartDocument,
		Depth:   0,
		Spaces:  map[string]string{},
	}
}

func (p *XMLPullParser) NextTag() (event XMLEventType, err error) {
	t, err := p.Next()
	if err != nil {
		return event, err
	}

	if t == Text && p.isWhitespace() {
		t, err = p.Next()
		if err != nil {
			return event, err
		}
	}

	if t != StartTag && t != EndTag {
		return event, fmt.Errorf("Expected StartTag or EndTag but got %s", p.eventName(t))
	}

	return t, nil
}

func (p *XMLPullParser) Next() (event XMLEventType, err error) {
	text := ""

	for {
		event, err = p.NextToken()
		if err != nil {
			return event, err
		}

		// Return immediately after encountering a StartTag
		// EndTag or EndDocument
		if event == StartTag ||
			event == EndTag ||
			event == EndDocument {
			return event, nil
		}

		if event == Comment ||
			event == Directive ||
			event == ProcessingInstruction {
			text = ""
			continue
		}

		// Coalesce text event value
		if event == Text {
			text += p.Text
		}

		// Return the text event if it is going to be
		// ended by a Start/EndTag or EndDocument.
		// Otherwise we will continue to coalesce text
		// events.
		if text != "" && (p.peekEvent == StartTag ||
			p.peekEvent == EndTag ||
			p.peekEvent == EndDocument) {
			p.Text = text
			return Text, nil
		}
	}
	return event, nil
}

func (p *XMLPullParser) NextToken() (event XMLEventType, err error) {
	// Clear any state held for the previous token
	p.resetTokenState()

	if p.peekErr != nil {
		return event, p.peekErr
	}

	if p.Event == StartDocument {
		// Preload the peek token when first starting
		p.peekNextToken()
	} else if p.Event == EndDocument {
		// Always return EndDocument once that state has
		// been reached.
		return p.Event, nil
	}

	// Switch peek token/event to the current token/event
	p.Event = p.peekEvent
	p.token = p.peekToken
	p.processToken(p.token)

	p.peekNextToken()

	// Return current event (previously the peek token)
	return p.Event, nil
}

func (p *XMLPullParser) peekNextToken() {
	// Peek the next token/event
	peekToken, err := p.decoder.Token()
	if err != nil {
		if err == io.EOF {
			// XML decoder returns the EOF as an error
			// but we want to return it as a valid
			// EndDocument token instead
			p.peekToken = nil
			p.peekEvent = EndDocument
		} else {
			p.peekErr = err
		}
		return
	}
	p.peekToken = xml.CopyToken(peekToken)
	p.peekEvent = p.eventType(peekToken)
}

func (p *XMLPullParser) NextText() (string, error) {
	if p.Event != StartTag {
		return "", errors.New("Parser must be on StartTag to get NextText()")
	}

	t, err := p.Next()
	if err != nil {
		return "", err
	}

	if t == Text {
		result := p.Text
		nt, err := p.Next()
		if err != nil {
			return "", err
		}

		if nt != EndTag {
			return "", errors.New("Event Text must be immediately followed by EndTag")
		}

		return result, nil
	} else if t == EndTag {
		return "", nil
	} else {
		return "", errors.New("Parser must be on StartTag or Text to read text")
	}
}

func (p *XMLPullParser) Skip() error {
	for {
		tok, err := p.NextToken()
		if err != nil {
			return err
		}
		if tok == StartTag {
			if err := p.Skip(); err != nil {
				return err
			}
		} else if tok == EndTag {
			return nil
		}
	}
}

func (p *XMLPullParser) Attribute(name string) string {
	for _, attr := range p.Attrs {
		if attr.Name.Local == name {
			return attr.Value
		}
	}
	return ""
}

func (p *XMLPullParser) Expect(event XMLEventType, name string) (err error) {
	return p.ExpectAll(event, "*", name)
}

func (p *XMLPullParser) ExpectAll(event XMLEventType, space string, name string) (err error) {
	if !(p.Event == event && (p.Space == space || space == "*") && (p.Name == name || name == "*")) {
		err = fmt.Errorf("Expected Space:%s Name:%s Event:%s but got Space:%s Name:%s Event:%s", space, name, p.eventName(event), p.Space, p.Name, p.eventName(p.Event))
	}
	return
}

func (p *XMLPullParser) DecodeElement(v interface{}) error {
	if p.Event != StartTag {
		return errors.New("DecodeElement can only be called from a StartTag event")
	}

	// Consumes all tokens until the matching end token.
	err := p.decoder.DecodeElement(v, p.token)
	if err != nil {
		return err
	}

	name := p.Name

	// Need to set the "current" token name/event
	// to the previous StartTag event's name
	p.resetTokenState()
	p.Event = EndTag
	p.Name = name
	p.token = nil

	// Need to peek the next token
	p.peekNextToken()
}

func (p *XMLPullParser) processToken(t xml.Token) {
	switch tt := t.(type) {
	case xml.StartElement:
		p.processStartToken(tt)
	case xml.EndElement:
		p.processEndToken(tt)
	case xml.CharData:
		p.processCharDataToken(tt)
	case xml.Comment:
		p.processCommentToken(tt)
	case xml.ProcInst:
		p.processProcInstToken(tt)
	case xml.Directive:
		p.processDirectiveToken(tt)
	}
}

func (p *XMLPullParser) processStartToken(t xml.StartElement) {
	p.Depth++
	p.Attrs = t.Attr
	p.Name = t.Name.Local
	p.Space = t.Name.Space
	p.trackNamespaces(t)
}

func (p *XMLPullParser) processEndToken(t xml.EndElement) {
	p.Depth--
	p.Name = t.Name.Local
}

func (p *XMLPullParser) processCharDataToken(t xml.CharData) {
	p.Text = string([]byte(t))
}

func (p *XMLPullParser) processCommentToken(t xml.Comment) {
	p.Text = string([]byte(t))
}

func (p *XMLPullParser) processProcInstToken(t xml.ProcInst) {
	p.Text = fmt.Sprintf("%s %s", t.Target, string(t.Inst))
}

func (p *XMLPullParser) processDirectiveToken(t xml.Directive) {
	p.Text = string([]byte(t))
}

func (p *XMLPullParser) resetTokenState() {
	p.Attrs = nil
	p.Name = ""
	p.Space = ""
	p.Text = ""
}

func (p *XMLPullParser) isWhitespace() bool {
	return strings.TrimSpace(p.Text) == ""
}

func (p *XMLPullParser) eventName(e XMLEventType) (name string) {
	switch e {
	case StartTag:
		name = "StartTag"
	case EndTag:
		name = "EndTag"
	case StartDocument:
		name = "StartDocument"
	case EndDocument:
		name = "EndDocument"
	case ProcessingInstruction:
		name = "ProcessingInstruction"
	case Directive:
		name = "Directive"
	case Comment:
		name = "Comment"
	case Text:
		name = "Text"
	case IgnorableWhitespace:
		name = "IgnorableWhitespace"
	}
	return
}

func (p *XMLPullParser) eventType(t xml.Token) (event XMLEventType) {
	switch t.(type) {
	case xml.StartElement:
		event = StartTag
	case xml.EndElement:
		event = EndTag
	case xml.CharData:
		event = Text
	case xml.Comment:
		event = Comment
	case xml.ProcInst:
		event = ProcessingInstruction
	case xml.Directive:
		event = Directive
	}
	return
}

func (p *XMLPullParser) trackNamespaces(t xml.StartElement) {
	for _, attr := range t.Attr {
		if attr.Name.Space == "xmlns" {
			space := strings.TrimSpace(attr.Value)
			spacePrefix := strings.TrimSpace(strings.ToLower(attr.Name.Local))
			p.Spaces[space] = spacePrefix
		} else if attr.Name.Local == "xmlns" {
			space := strings.TrimSpace(attr.Value)
			p.Spaces[space] = ""
		}
	}
}
<|endoftext|>


<commit_before>package nsf

type Apu struct {
	S1, S2 Square
	Triangle

	Odd        bool
	FC         byte
	FT         byte
	IrqDisable bool
}

type Triangle struct {
	Linear
	Timer
	Length
	SI int // sequence index

	Enable bool
}

type Linear struct {
	Reload  byte
	Halt    bool
	Flag    bool
	Counter byte
}

type Square struct {
	Envelope
	Timer
	Length
	Sweep
	Duty

	Enable bool
}

type Duty struct {
	Type    byte
	Counter byte
}

type Sweep struct {
	Shift     byte
	Negate    bool
	Period    byte
	Enable    bool
	Divider   byte
	Reset     bool
	NegOffset int
}

type Envelope struct {
	Volume   byte
	Divider  byte
	Counter  byte
	Loop     bool
	Constant bool
	Start    bool
}

type Timer struct {
	Tick   uint16
	Length uint16
}

type Length struct {
	Halt    bool
	Counter byte
}

func (a *Apu) Init() {
	a.S1.Sweep.NegOffset = -1
	for i := uint16(0x4000); i <= 0x400f; i++ {
		a.Write(i, 0)
	}
	a.Write(0x4010, 0x10)
	a.Write(0x4011, 0)
	a.Write(0x4012, 0)
	a.Write(0x4013, 0)
	a.Write(0x4015, 0xf)
	a.Write(0x4017, 0)
}

func (a *Apu) Write(v uint16, b byte) {
	switch v & 0xff {
	case 0x00:
		a.S1.Control1(b)
	case 0x01:
		a.S1.Control2(b)
	case 0x02:
		a.S1.Control3(b)
	case 0x03:
		a.S1.Control4(b)
	case 0x04:
		a.S2.Control1(b)
	case 0x05:
		a.S2.Control2(b)
	case 0x06:
		a.S2.Control3(b)
	case 0x07:
		a.S2.Control4(b)
	case 0x08:
		a.Triangle.Control1(b)
	case 0x0a:
		a.Triangle.Control2(b)
	case 0x0b:
		a.Triangle.Control3(b)
	case 0x15:
		a.S1.Disable(b&0x1 == 0)
		a.S2.Disable(b&0x2 == 0)
		a.Triangle.Disable(b&0x4 == 0)
	case 0x17:
		a.FT = 0
		if b&0x80 != 0 {
			a.FC = 5
			a.FrameStep()
		} else {
			a.FC = 4
		}
		a.IrqDisable = b&0x40 != 0
	}
}

func (t *Triangle) Control1(b byte) {
	t.Linear.Control(b)
	t.Length.Halt = b&0x80 != 0
}

func (l *Linear) Control(b byte) {
	l.Flag = b&0x80 != 0
	l.Reload = b & 0x7f
}

func (t *Triangle) Control2(b byte) {
	t.Timer.Length &= 0xff00
	t.Timer.Length |= uint16(b)
}

func (t *Triangle) Control3(b byte) {
	t.Timer.Length &= 0xff
	t.Timer.Length |= uint16(b&0x7) << 8
	t.Length.Set(b >> 3)
	t.Linear.Halt = true
}

func (s *Square) Control1(b byte) {
	s.Envelope.Control(b)
	s.Duty.Control(b)
	s.Length.Halt = b&0x20 != 0
}

func (s *Square) Control2(b byte) {
	s.Sweep.Control(b)
}

func (s *Square) Control3(b byte) {
	s.Timer.Length &= 0xff00
	s.Timer.Length |= uint16(b)
}

func (s *Square) Control4(b byte) {
	s.Timer.Length &= 0xff
	s.Timer.Length |= uint16(b&0x7) << 8
	s.Length.Set(b >> 3)

	s.Envelope.Start = true
	s.Duty.Counter = 0
}

func (d *Duty) Control(b byte) {
	d.Type = b >> 6
}

func (s *Sweep) Control(b byte) {
	s.Shift = b & 0x7
	s.Negate = b&0x8 != 0
	s.Period = (b >> 4) & 0x7
	s.Enable = b&0x80 != 0
	s.Reset = true
}

func (e *Envelope) Control(b byte) {
	e.Volume = b & 0xf
	e.Constant = b&0x10 != 0
	e.Loop = b&0x20 != 0
}

func (l *Length) Set(b byte) {
	if !l.Halt {
		l.Counter = LenLookup[b]
	}
}

func (l *Length) Enabled() bool {
	return l.Counter != 0
}

func (s *Square) Disable(b bool) {
	s.Enable = !b
	if b {
		s.Length.Counter = 0
	}
}

func (t *Triangle) Disable(b bool) {
	t.Enable = !b
	if b {
		t.Length.Counter = 0
	}
}

func (a *Apu) Read(v uint16) byte {
	var b byte
	if v == 0x4015 {
		if a.S1.Length.Counter > 0 {
			b |= 0x1
		}
		if a.S2.Length.Counter > 0 {
			b |= 0x2
		}
		if a.Triangle.Length.Counter > 0 {
			b |= 0x4
		}
	}
	return b
}

func (d *Duty) Clock() {
	if d.Counter == 0 {
		d.Counter = 7
	} else {
		d.Counter--
	}
}

func (s *Sweep) Clock() (r bool) {
	if s.Divider == 0 {
		s.Divider = s.Period
		r = true
	} else {
		s.Divider--
	}
	if s.Reset {
		s.Divider = 0
		s.Reset = false
	}
	return
}

func (e *Envelope) Clock() {
	if e.Start {
		e.Start = false
		e.Counter = 15
	} else {
		if e.Divider == 0 {
			e.Divider = e.Volume
			if e.Counter != 0 {
				e.Counter--
			} else if e.Loop {
				e.Counter = 15
			}
		} else {
			e.Divider--
		}
	}
}

func (t *Timer) Clock() bool {
	if t.Tick == 0 {
		t.Tick = t.Length
	} else {
		t.Tick--
	}
	return t.Tick == t.Length
}

func (s *Square) Clock() {
	if s.Timer.Clock() {
		s.Duty.Clock()
	}
}

func (t *Triangle) Clock() {
	if t.Timer.Clock() && t.Length.Counter > 0 && t.Linear.Counter > 0 {
		if t.SI == 31 {
			t.SI = 0
		} else {
			t.SI++
		}
	}
}

func (a *Apu) Step() {
	if a.Odd {
		if a.S1.Enable {
			a.S1.Clock()
		}
		if a.S2.Enable {
			a.S2.Clock()
		}
	}
	a.Odd = !a.Odd
	if a.Triangle.Enable {
		a.Triangle.Clock()
	}
}

func (a *Apu) FrameStep() {
	a.FT++
	if a.FT == a.FC {
		a.FT = 0
	}
	if a.FT <= 3 {
		a.S1.Envelope.Clock()
		a.Triangle.Linear.Clock()
	}
	if a.FT == 1 || a.FT == 3 {
		a.S1.FrameStep()
		a.S2.FrameStep()
		a.Triangle.Length.Clock()
	}
	if a.FC == 4 && a.FT == 3 && !a.IrqDisable {
		// todo: assert cpu irq line
	}
}

func (l *Linear) Clock() {
	if l.Halt {
		l.Counter = l.Reload
	} else if l.Counter != 0 {
		l.Counter--
	}
	if !l.Flag {
		l.Halt = false
	}
}

func (s *Square) FrameStep() {
	s.Length.Clock()
	if s.Sweep.Clock() && s.Sweep.Enable && s.Sweep.Shift > 0 {
		r := s.SweepResult()
		if r <= 0x7ff {
			s.Timer.Tick = r
		}
	}
}

func (l *Length) Clock() {
	if !l.Halt && l.Counter > 0 {
		l.Counter--
	}
}

func (a *Apu) Volume() float32 {
	p := PulseOut[a.S1.Volume()+a.S2.Volume()]
	t := TndOut[3*a.Triangle.Volume()]
	return p + t
}

func (t *Triangle) Volume() uint8 {
	if t.Enable && t.Linear.Counter > 0 && t.Length.Counter > 0 {
		return TriLookup[t.SI]
	}
	return 0
}

func (s *Square) Volume() uint8 {
	if s.Enable && s.Duty.Enabled() && s.Length.Enabled() && s.Timer.Tick >= 8 && s.SweepResult() <= 0x7ff {
		return s.Envelope.Output()
	}
	return 0
}

func (e *Envelope) Output() byte {
	if e.Constant {
		return e.Volume
	}
	return e.Counter
}

func (s *Square) SweepResult() uint16 {
	r := int(s.Timer.Tick >> s.Sweep.Shift)
	if s.Sweep.Negate {
		r = -r
	}
	r += int(s.Timer.Tick)
	if r > 0x7ff {
		r = 0x800
	}
	return uint16(r)
}

func (d *Duty) Enabled() bool {
	return DutyCycle[d.Type][d.Counter] == 1
}

var (
	PulseOut  [31]float32
	TndOut    [203]float32
	DutyCycle = [4][8]byte{
		{0, 1, 0, 0, 0, 0, 0, 0},
		{0, 1, 1, 0, 0, 0, 0, 0},
		{0, 1, 1, 1, 1, 0, 0, 0},
		{1, 0, 0, 1, 1, 1, 1, 1},
	}
	LenLookup = [...]byte{
		0x0a, 0xfe, 0x14, 0x02,
		0x28, 0x04, 0x50, 0x06,
		0xa0, 0x08, 0x3c, 0x0a,
		0x0e, 0x0c, 0x1a, 0x0e,
		0x0c, 0x10, 0x18, 0x12,
		0x30, 0x14, 0x60, 0x16,
		0xc0, 0x18, 0x48, 0x1a,
		0x10, 0x1c, 0x20, 0x1e,
	}
	TriLookup = [...]byte{
		0xF, 0xE, 0xD, 0xC,
		0xB, 0xA, 0x9, 0x8,
		0x7, 0x6, 0x5, 0x4,
		0x3, 0x2, 0x1, 0x0,
		0x0, 0x1, 0x2, 0x3,
		0x4, 0x5, 0x6, 0x7,
		0x8, 0x9, 0xA, 0xB,
		0xC, 0xD, 0xE, 0xF,
	}
)

func init() {
	for i := range PulseOut {
		PulseOut[i] = 95.88 / (8128/float32(i) + 100)
	}
	for i := range TndOut {
		TndOut[i] = 163.67 / (24329/float32(i) + 100)
	}
}
<commit_msg>Allow setting of length counter when halted<commit_after>package nsf

type Apu struct {
	S1, S2 Square
	Triangle

	Odd        bool
	FC         byte
	FT         byte
	IrqDisable bool
}

type Triangle struct {
	Linear
	Timer
	Length
	SI int // sequence index

	Enable bool
}

type Linear struct {
	Reload  byte
	Halt    bool
	Flag    bool
	Counter byte
}

type Square struct {
	Envelope
	Timer
	Length
	Sweep
	Duty

	Enable bool
}

type Duty struct {
	Type    byte
	Counter byte
}

type Sweep struct {
	Shift     byte
	Negate    bool
	Period    byte
	Enable    bool
	Divider   byte
	Reset     bool
	NegOffset int
}

type Envelope struct {
	Volume   byte
	Divider  byte
	Counter  byte
	Loop     bool
	Constant bool
	Start    bool
}

type Timer struct {
	Tick   uint16
	Length uint16
}

type Length struct {
	Halt    bool
	Counter byte
}

func (a *Apu) Init() {
	a.S1.Sweep.NegOffset = -1
	for i := uint16(0x4000); i <= 0x400f; i++ {
		a.Write(i, 0)
	}
	a.Write(0x4010, 0x10)
	a.Write(0x4011, 0)
	a.Write(0x4012, 0)
	a.Write(0x4013, 0)
	a.Write(0x4015, 0xf)
	a.Write(0x4017, 0)
}

func (a *Apu) Write(v uint16, b byte) {
	switch v & 0xff {
	case 0x00:
		a.S1.Control1(b)
	case 0x01:
		a.S1.Control2(b)
	case 0x02:
		a.S1.Control3(b)
	case 0x03:
		a.S1.Control4(b)
	case 0x04:
		a.S2.Control1(b)
	case 0x05:
		a.S2.Control2(b)
	case 0x06:
		a.S2.Control3(b)
	case 0x07:
		a.S2.Control4(b)
	case 0x08:
		a.Triangle.Control1(b)
	case 0x0a:
		a.Triangle.Control2(b)
	case 0x0b:
		a.Triangle.Control3(b)
	case 0x15:
		a.S1.Disable(b&0x1 == 0)
		a.S2.Disable(b&0x2 == 0)
		a.Triangle.Disable(b&0x4 == 0)
	case 0x17:
		a.FT = 0
		if b&0x80 != 0 {
			a.FC = 5
			a.FrameStep()
		} else {
			a.FC = 4
		}
		a.IrqDisable = b&0x40 != 0
	}
}

func (t *Triangle) Control1(b byte) {
	t.Linear.Control(b)
	t.Length.Halt = b&0x80 != 0
}

func (l *Linear) Control(b byte) {
	l.Flag = b&0x80 != 0
	l.Reload = b & 0x7f
}

func (t *Triangle) Control2(b byte) {
	t.Timer.Length &= 0xff00
	t.Timer.Length |= uint16(b)
}

func (t *Triangle) Control3(b byte) {
	t.Timer.Length &= 0xff
	t.Timer.Length |= uint16(b&0x7) << 8
	t.Length.Set(b >> 3)
	t.Linear.Halt = true
}

func (s *Square) Control1(b byte) {
	s.Envelope.Control(b)
	s.Duty.Control(b)
	s.Length.Halt = b&0x20 != 0
}

func (s *Square) Control2(b byte) {
	s.Sweep.Control(b)
}

func (s *Square) Control3(b byte) {
	s.Timer.Length &= 0xff00
	s.Timer.Length |= uint16(b)
}

func (s *Square) Control4(b byte) {
	s.Timer.Length &= 0xff
	s.Timer.Length |= uint16(b&0x7) << 8
	s.Length.Set(b >> 3)

	s.Envelope.Start = true
	s.Duty.Counter = 0
}

func (d *Duty) Control(b byte) {
	d.Type = b >> 6
}

func (s *Sweep) Control(b byte) {
	s.Shift = b & 0x7
	s.Negate = b&0x8 != 0
	s.Period = (b >> 4) & 0x7
	s.Enable = b&0x80 != 0
	s.Reset = true
}

func (e *Envelope) Control(b byte) {
	e.Volume = b & 0xf
	e.Constant = b&0x10 != 0
	e.Loop = b&0x20 != 0
}

func (l *Length) Set(b byte) {
	l.Counter = LenLookup[b]
}

func (l *Length) Enabled() bool {
	return l.Counter != 0
}

func (s *Square) Disable(b bool) {
	s.Enable = !b
	if b {
		s.Length.Counter = 0
	}
}

func (t *Triangle) Disable(b bool) {
	t.Enable = !b
	if b {
		t.Length.Counter = 0
	}
}

func (a *Apu) Read(v uint16) byte {
	var b byte
	if v == 0x4015 {
		if a.S1.Length.Counter > 0 {
			b |= 0x1
		}
		if a.S2.Length.Counter > 0 {
			b |= 0x2
		}
		if a.Triangle.Length.Counter > 0 {
			b |= 0x4
		}
	}
	return b
}

func (d *Duty) Clock() {
	if d.Counter == 0 {
		d.Counter = 7
	} else {
		d.Counter--
	}
}

func (s *Sweep) Clock() (r bool) {
	if s.Divider == 0 {
		s.Divider = s.Period
		r = true
	} else {
		s.Divider--
	}
	if s.Reset {
		s.Divider = 0
		s.Reset = false
	}
	return
}

func (e *Envelope) Clock() {
	if e.Start {
		e.Start = false
		e.Counter = 15
	} else {
		if e.Divider == 0 {
			e.Divider = e.Volume
			if e.Counter != 0 {
				e.Counter--
			} else if e.Loop {
				e.Counter = 15
			}
		} else {
			e.Divider--
		}
	}
}

func (t *Timer) Clock() bool {
	if t.Tick == 0 {
		t.Tick = t.Length
	} else {
		t.Tick--
	}
	return t.Tick == t.Length
}

func (s *Square) Clock() {
	if s.Timer.Clock() {
		s.Duty.Clock()
	}
}

func (t *Triangle) Clock() {
	if t.Timer.Clock() && t.Length.Counter > 0 && t.Linear.Counter > 0 {
		if t.SI == 31 {
			t.SI = 0
		} else {
			t.SI++
		}
	}
}

func (a *Apu) Step() {
	if a.Odd {
		if a.S1.Enable {
			a.S1.Clock()
		}
		if a.S2.Enable {
			a.S2.Clock()
		}
	}
	a.Odd = !a.Odd
	if a.Triangle.Enable {
		a.Triangle.Clock()
	}
}

func (a *Apu) FrameStep() {
	a.FT++
	if a.FT == a.FC {
		a.FT = 0
	}
	if a.FT <= 3 {
		a.S1.Envelope.Clock()
		a.Triangle.Linear.Clock()
	}
	if a.FT == 1 || a.FT == 3 {
		a.S1.FrameStep()
		a.S2.FrameStep()
		a.Triangle.Length.Clock()
	}
	if a.FC == 4 && a.FT == 3 && !a.IrqDisable {
		// todo: assert cpu irq line
	}
}

func (l *Linear) Clock() {
	if l.Halt {
		l.Counter = l.Reload
	} else if l.Counter != 0 {
		l.Counter--
	}
	if !l.Flag {
		l.Halt = false
	}
}

func (s *Square) FrameStep() {
	s.Length.Clock()
	if s.Sweep.Clock() && s.Sweep.Enable && s.Sweep.Shift > 0 {
		r := s.SweepResult()
		if r <= 0x7ff {
			s.Timer.Tick = r
		}
	}
}

func (l *Length) Clock() {
	if !l.Halt && l.Counter > 0 {
		l.Counter--
	}
}

func (a *Apu) Volume() float32 {
	p := PulseOut[a.S1.Volume()+a.S2.Volume()]
	t := TndOut[3*a.Triangle.Volume()]
	return p + t
}

func (t *Triangle) Volume() uint8 {
	if t.Enable && t.Linear.Counter > 0 && t.Length.Counter > 0 {
		return TriLookup[t.SI]
	}
	return 0
}

func (s *Square) Volume() uint8 {
	if s.Enable && s.Duty.Enabled() && s.Length.Enabled() && s.Timer.Tick >= 8 && s.SweepResult() <= 0x7ff {
		return s.Envelope.Output()
	}
	return 0
}

func (e *Envelope) Output() byte {
	if e.Constant {
		return e.Volume
	}
	return e.Counter
}

func (s *Square) SweepResult() uint16 {
	r := int(s.Timer.Tick >> s.Sweep.Shift)
	if s.Sweep.Negate {
		r = -r
	}
	r += int(s.Timer.Tick)
	if r > 0x7ff {
		r = 0x800
	}
	return uint16(r)
}

func (d *Duty) Enabled() bool {
	return DutyCycle[d.Type][d.Counter] == 1
}

var (
	PulseOut  [31]float32
	TndOut    [203]float32
	DutyCycle = [4][8]byte{
		{0, 1, 0, 0, 0, 0, 0, 0},
		{0, 1, 1, 0, 0, 0, 0, 0},
		{0, 1, 1, 1, 1, 0, 0, 0},
		{1, 0, 0, 1, 1, 1, 1, 1},
	}
	LenLookup = [...]byte{
		0x0a, 0xfe, 0x14, 0x02,
		0x28, 0x04, 0x50, 0x06,
		0xa0, 0x08, 0x3c, 0x0a,
		0x0e, 0x0c, 0x1a, 0x0e,
		0x0c, 0x10, 0x18, 0x12,
		0x30, 0x14, 0x60, 0x16,
		0xc0, 0x18, 0x48, 0x1a,
		0x10, 0x1c, 0x20, 0x1e,
	}
	TriLookup = [...]byte{
		0xF, 0xE, 0xD, 0xC,
		0xB, 0xA, 0x9, 0x8,
		0x7, 0x6, 0x5, 0x4,
		0x3, 0x2, 0x1, 0x0,
		0x0, 0x1, 0x2, 0x3,
		0x4, 0x5, 0x6, 0x7,
		0x8, 0x9, 0xA, 0xB,
		0xC, 0xD, 0xE, 0xF,
	}
)

func init() {
	for i := range PulseOut {
		PulseOut[i] = 95.88 / (8128/float32(i) + 100)
	}
	for i := range TndOut {
		TndOut[i] = 163.67 / (24329/float32(i) + 100)
	}
}
<|endoftext|>


<commit_before>package network

import (
	context "github.com/jbenet/go-ipfs/Godeps/_workspace/src/code.google.com/p/go.net/context"

	bsmsg "github.com/jbenet/go-ipfs/exchange/bitswap/message"
	inet "github.com/jbenet/go-ipfs/net"
	netmsg "github.com/jbenet/go-ipfs/net/message"
	peer "github.com/jbenet/go-ipfs/peer"
)

// NetMessageAdapter wraps a NetMessage network service
func NetMessageAdapter(s inet.Service, n inet.Network, r Receiver) Adapter {
	adapter := impl{
		nms:      s,
		net:      n,
		receiver: r,
	}
	s.SetHandler(&adapter)
	return &adapter
}

// implements an Adapter that integrates with a NetMessage network service
type impl struct {
	nms inet.Service
	net inet.Network

	// inbound messages from the network are forwarded to the receiver
	receiver Receiver
}

// HandleMessage marshals and unmarshals net messages, forwarding them to the
// BitSwapMessage receiver
func (adapter *impl) HandleMessage(
	ctx context.Context, incoming netmsg.NetMessage) netmsg.NetMessage {

	if adapter.receiver == nil {
		return nil
	}

	received, err := bsmsg.FromNet(incoming)
	if err != nil {
		go adapter.receiver.ReceiveError(err)
		return nil
	}

	p, bsmsg := adapter.receiver.ReceiveMessage(ctx, incoming.Peer(), received)

	// TODO(brian): put this in a helper function
	if bsmsg == nil || p == nil {
		return nil
	}

	outgoing, err := bsmsg.ToNet(p)
	if err != nil {
		go adapter.receiver.ReceiveError(err)
		return nil
	}

	return outgoing
}

func (adapter *impl) DialPeer(p *peer.Peer) error {
	return adapter.DialPeer(p)
}

func (adapter *impl) SendMessage(
	ctx context.Context,
	p *peer.Peer,
	outgoing bsmsg.BitSwapMessage) error {

	nmsg, err := outgoing.ToNet(p)
	if err != nil {
		return err
	}
	return adapter.nms.SendMessage(ctx, nmsg)
}

func (adapter *impl) SendRequest(
	ctx context.Context,
	p *peer.Peer,
	outgoing bsmsg.BitSwapMessage) (bsmsg.BitSwapMessage, error) {

	outgoingMsg, err := outgoing.ToNet(p)
	if err != nil {
		return nil, err
	}
	incomingMsg, err := adapter.nms.SendRequest(ctx, outgoingMsg)
	if err != nil {
		return nil, err
	}
	return bsmsg.FromNet(incomingMsg)
}

func (adapter *impl) SetDelegate(r Receiver) {
	adapter.receiver = r
}
<commit_msg>meant to call net.DialPeer<commit_after>package network

import (
	context "github.com/jbenet/go-ipfs/Godeps/_workspace/src/code.google.com/p/go.net/context"

	bsmsg "github.com/jbenet/go-ipfs/exchange/bitswap/message"
	inet "github.com/jbenet/go-ipfs/net"
	netmsg "github.com/jbenet/go-ipfs/net/message"
	peer "github.com/jbenet/go-ipfs/peer"
)

// NetMessageAdapter wraps a NetMessage network service
func NetMessageAdapter(s inet.Service, n inet.Network, r Receiver) Adapter {
	adapter := impl{
		nms:      s,
		net:      n,
		receiver: r,
	}
	s.SetHandler(&adapter)
	return &adapter
}

// implements an Adapter that integrates with a NetMessage network service
type impl struct {
	nms inet.Service
	net inet.Network

	// inbound messages from the network are forwarded to the receiver
	receiver Receiver
}

// HandleMessage marshals and unmarshals net messages, forwarding them to the
// BitSwapMessage receiver
func (adapter *impl) HandleMessage(
	ctx context.Context, incoming netmsg.NetMessage) netmsg.NetMessage {

	if adapter.receiver == nil {
		return nil
	}

	received, err := bsmsg.FromNet(incoming)
	if err != nil {
		go adapter.receiver.ReceiveError(err)
		return nil
	}

	p, bsmsg := adapter.receiver.ReceiveMessage(ctx, incoming.Peer(), received)

	// TODO(brian): put this in a helper function
	if bsmsg == nil || p == nil {
		return nil
	}

	outgoing, err := bsmsg.ToNet(p)
	if err != nil {
		go adapter.receiver.ReceiveError(err)
		return nil
	}

	return outgoing
}

func (adapter *impl) DialPeer(p *peer.Peer) error {
	return adapter.net.DialPeer(p)
}

func (adapter *impl) SendMessage(
	ctx context.Context,
	p *peer.Peer,
	outgoing bsmsg.BitSwapMessage) error {

	nmsg, err := outgoing.ToNet(p)
	if err != nil {
		return err
	}
	return adapter.nms.SendMessage(ctx, nmsg)
}

func (adapter *impl) SendRequest(
	ctx context.Context,
	p *peer.Peer,
	outgoing bsmsg.BitSwapMessage) (bsmsg.BitSwapMessage, error) {

	outgoingMsg, err := outgoing.ToNet(p)
	if err != nil {
		return nil, err
	}
	incomingMsg, err := adapter.nms.SendRequest(ctx, outgoingMsg)
	if err != nil {
		return nil, err
	}
	return bsmsg.FromNet(incomingMsg)
}

func (adapter *impl) SetDelegate(r Receiver) {
	adapter.receiver = r
}
<|endoftext|>


<commit_before>package main

import (
	"flag"
	"fmt"
	"github.com/APTrust/exchange/context"
	"github.com/APTrust/exchange/models"
	"github.com/APTrust/exchange/workers"
	"os"
	"time"
)

// 20 GB
const MAX_FILE_SIZE = int64(21474836480)
const DAYS_SINCE_INGEST = 180
const DAYS_SINCE_LAST_RESTORE = 180

func main() {
	pathToConfigFile, dryRun := parseCommandLine()
	config, err := models.LoadConfigFile(pathToConfigFile)
	if err != nil {
		fmt.Fprintf(os.Stderr, err.Error())
		os.Exit(1)
	}
	createdBefore := time.Now().UTC().AddDate(0, 0, DAYS_SINCE_INGEST)
	notRestoredSince := time.Now().UTC().AddDate(0, 0, DAYS_SINCE_LAST_RESTORE)
	_context := context.NewContext(config)
	worker := workers.NewAPTSpotTestRestore(_context, MAX_FILE_SIZE, createdBefore, notRestoredSince)
	worker.DryRun = dryRun
	items, err := worker.Run()
	if err != nil {
		fmt.Fprintln(os.Stderr, "Error: ", err.Error())
	} else {
		printResults(items)
	}
}

func printResults(items []*models.WorkItem) {
	fmt.Println("Created Restore WorkItems for the following intellectual objects:")
	for _, item := range items {
		fmt.Println(item.ObjectIdentifier)
	}
}

// See if you can figure out from the function name what this does.
func parseCommandLine() (string, bool) {
	var pathToConfigFile string
	dryRun := flag.Bool("dryrun", false, "List which bags would be chosen, but don't queue any WorkItems")
	flag.StringVar(&pathToConfigFile, "config", "", "Path to APTrust config file")
	flag.Parse()
	if pathToConfigFile == "" {
		printUsage()
		os.Exit(1)
	}
	return pathToConfigFile, *dryRun
}

// Tell the user about the program.
func printUsage() {
	message := `

apt_spot_test_restore selects one bag from each institution to restore, so that
APTrust and its depositors can see that bag restoration is working properly.
After selecting a bag, this app creates a Restore WorkItem in Pharos for the
bag. From there, it follows the normal restoration process.

The restorer chooses the first bag from each institution that meets the
following criteria:

* Bag is less than 20 GB in size.
* Bag was ingested at least 180 days ago.
* Bag has not been restored in the past 180 days.

Usage:

    apt_spot_test_restore -config=<absolute path to APTrust config file>
    apt_spot_test_restore -config=<absolute path to APTrust config file> -dryrun

Param -config is required.

If param -dryrun is present, the program will return a list of bags that would be
queued for restoration, but it won't actually queue them.

`
	fmt.Println(message)
}
<commit_msg>Fixed https://trello.com/c/dMoQJnnT. Choose items that have not been restored in PAST 180 days.<commit_after>package main

import (
	"flag"
	"fmt"
	"github.com/APTrust/exchange/context"
	"github.com/APTrust/exchange/models"
	"github.com/APTrust/exchange/workers"
	"os"
	"time"
)

// 53687091200 = 50 GB
const MAX_FILE_SIZE = int64(53687091200)
const DAYS_SINCE_INGEST = 180
const DAYS_SINCE_LAST_RESTORE = 180

func main() {
	pathToConfigFile, dryRun := parseCommandLine()
	config, err := models.LoadConfigFile(pathToConfigFile)
	if err != nil {
		fmt.Fprintf(os.Stderr, err.Error())
		os.Exit(1)
	}
	createdBefore := time.Now().UTC().AddDate(0, 0, (-1 * DAYS_SINCE_INGEST))
	notRestoredSince := time.Now().UTC().AddDate(0, 0, (-1 * DAYS_SINCE_LAST_RESTORE))
	_context := context.NewContext(config)
	worker := workers.NewAPTSpotTestRestore(_context, MAX_FILE_SIZE, createdBefore, notRestoredSince)
	worker.DryRun = dryRun
	items, err := worker.Run()
	if err != nil {
		fmt.Fprintln(os.Stderr, "Error: ", err.Error())
	} else {
		printResults(items)
	}
}

func printResults(items []*models.WorkItem) {
	fmt.Println("Created Restore WorkItems for the following intellectual objects:")
	for _, item := range items {
		fmt.Println(item.ObjectIdentifier)
	}
}

// See if you can figure out from the function name what this does.
func parseCommandLine() (string, bool) {
	var pathToConfigFile string
	dryRun := flag.Bool("dryrun", false, "List which bags would be chosen, but don't queue any WorkItems")
	flag.StringVar(&pathToConfigFile, "config", "", "Path to APTrust config file")
	flag.Parse()
	if pathToConfigFile == "" {
		printUsage()
		os.Exit(1)
	}
	return pathToConfigFile, *dryRun
}

// Tell the user about the program.
func printUsage() {
	message := `

apt_spot_test_restore selects one bag from each institution to restore, so that
APTrust and its depositors can see that bag restoration is working properly.
After selecting a bag, this app creates a Restore WorkItem in Pharos for the
bag. From there, it follows the normal restoration process.

The restorer chooses the first bag from each institution that meets the
following criteria:

* Bag is less than 20 GB in size.
* Bag was ingested at least 180 days ago.
* Bag has not been restored in the past 180 days.

Usage:

    apt_spot_test_restore -config=<absolute path to APTrust config file>
    apt_spot_test_restore -config=<absolute path to APTrust config file> -dryrun

Param -config is required.

If param -dryrun is present, the program will return a list of bags that would be
queued for restoration, but it won't actually queue them.

`
	fmt.Println(message)
}
<|endoftext|>


<commit_before>package consensus

import (
	"github.com/golang/glog"
	"github.com/heidi-ann/ios/msgs"
	"time"
)

var noop = msgs.ClientRequest{-1, -1, true, "noop"}

// RunMaster implements the Master mode
func RunMaster(view int, commit_index int, initial bool, io *msgs.Io, config Config) {
	// setup
	glog.Info("Starting up master in view ", view)
	glog.Info("Master is configured to delegate replication to ",config.DelegateReplication)
	majority := Majority(config.N)

	// determine next safe index
	index := -1
	if !initial {
		// dispatch new view requests
		req := msgs.NewViewRequest{config.ID, view}
		(*io).OutgoingBroadcast.Requests.NewView <- req

		// collect responses
		glog.Info("Waiting for ", majority, " new view responses")
		min_index := commit_index
		// TODO: FEATURE add option to wait longer

		for i := 0; i < majority; {
			msg := <-(*io).Incoming.Responses.NewView
			// check msg replies to the msg we just sent
			if msg.Request == req {
				res := msg.Response
				glog.Info("Received ", res)
				if res.Index > index {
					index = res.Index
				} else if res.Index < min_index {
					min_index = res.Index
				}
				i++
				// TODO: BUG need to check view
				glog.Info("Successful new view received, waiting for ", majority-i, " more")
			}

		}
		glog.Info("Index is ", index)

		// recover entries
		for curr_index := commit_index + 1; curr_index <= index; curr_index++ {
			RunRecoveryCoordinator(view, curr_index, io, config)
		}

	}
	// store the first coordinator to ask
	coordinator := config.ID
	if config.DelegateReplication > 0 {
		coordinator += 1
	}

	if config.BatchInterval == 0 {
		glog.Info("Ready to handle requests. No batching enabled")
		// handle client requests (1 at a time)
		for {

			// wait for request
			req := <-io.IncomingRequests
			glog.Info("Request received: ", req)

			// if possible, handle request without replication
			if !req.Replicate {
				io.OutgoingRequests <- req
				glog.Info("Read-only request was handled without replication: ", req)
			} else {
				index++
				glog.Info("Request assigned index: ", index)
				// ok := RunCoordinator(view, index, []msgs.ClientRequest{req}, io, config, true)
				entry := msgs.Entry{view, false, []msgs.ClientRequest{req}}
				coord := msgs.CoordinateRequest{config.ID, view, index, true, entry}
				io.OutgoingUnicast[coordinator].Requests.Coordinate <- coord
				// TODO: BUG: need to handle coordinator failure
				go func() {
					reply := <-(*io).Incoming.Responses.Coordinate
					// TODO: check msg replies to the msg we just sent
					if !reply.Response.Success {
						glog.Warning("Commit unsuccessful")
					}
					glog.Info("Finished replicating request: ", req)
				}()

				// rotate coordinator is nessacary
				if config.DelegateReplication > 1 {
					coordinator += 1
					if coordinator>config.ID + config.DelegateReplication {
						coordinator = config.ID + 1
					}
				}
			}

		}
	} else {
		glog.Info("Ready to handle request. Batch every ", config.BatchInterval, " microseconds")
		for {
			// setup for holding requests
			reqs := make([]msgs.ClientRequest, config.MaxBatch)
			reqs_num := 0

			// start collecting requests
			timeout := make(chan bool, 1)
			go func() {
				<-time.After(time.Microsecond * time.Duration(config.BatchInterval))
				timeout <- true
			}()

			exit := false
			for exit == false {
				select {
				case req := <-io.IncomingRequests:
					if !req.Replicate {
						io.OutgoingRequests <- req
						glog.Info("Request handled without replication: ", req)
					} else {
						reqs[reqs_num] = req
						glog.Info("Request ", reqs_num, " is : ", req)
						reqs_num = reqs_num + 1
						if reqs_num == config.MaxBatch {
							exit = true
							break
						}
					}
				case <-timeout:
					exit = true
					break
				}
			}

			// assign requests to coordinators
			if reqs_num > 0 {
				glog.Info("Starting to replicate ", reqs_num, " requests")
				reqs_small := reqs[:reqs_num]
				index++
				// ok := RunCoordinator(view, index, reqs_small, io, config, true)
				entry := msgs.Entry{view, false, reqs_small}
				coord := msgs.CoordinateRequest{config.ID, view, index, true, entry}
				io.OutgoingUnicast[coordinator].Requests.Coordinate <- coord
				go func() {
					reply := <- io.Incoming.Responses.Coordinate
					// TODO: check msg replies to the msg we just sent
					if !reply.Response.Success {
						glog.Warning("Commit unsuccessful")
					}
					glog.Info("Finished replicating requests: ", reqs_small)
				} ()
				// rotate coordinator is nessacary
				if config.DelegateReplication > 1 {
					coordinator += 1
					if coordinator>config.ID + config.DelegateReplication {
						coordinator = config.ID + 1
					}
				}
			}
		}

	}

	glog.Warning("Master stepping down")

}
<commit_msg>switching batching interval to milliseconds<commit_after>package consensus

import (
	"github.com/golang/glog"
	"github.com/heidi-ann/ios/msgs"
	"time"
)

var noop = msgs.ClientRequest{-1, -1, true, "noop"}

// RunMaster implements the Master mode
func RunMaster(view int, commit_index int, initial bool, io *msgs.Io, config Config) {
	// setup
	glog.Info("Starting up master in view ", view)
	glog.Info("Master is configured to delegate replication to ",config.DelegateReplication)
	majority := Majority(config.N)

	// determine next safe index
	index := -1
	if !initial {
		// dispatch new view requests
		req := msgs.NewViewRequest{config.ID, view}
		(*io).OutgoingBroadcast.Requests.NewView <- req

		// collect responses
		glog.Info("Waiting for ", majority, " new view responses")
		min_index := commit_index
		// TODO: FEATURE add option to wait longer

		for i := 0; i < majority; {
			msg := <-(*io).Incoming.Responses.NewView
			// check msg replies to the msg we just sent
			if msg.Request == req {
				res := msg.Response
				glog.Info("Received ", res)
				if res.Index > index {
					index = res.Index
				} else if res.Index < min_index {
					min_index = res.Index
				}
				i++
				// TODO: BUG need to check view
				glog.Info("Successful new view received, waiting for ", majority-i, " more")
			}

		}
		glog.Info("Index is ", index)

		// recover entries
		for curr_index := commit_index + 1; curr_index <= index; curr_index++ {
			RunRecoveryCoordinator(view, curr_index, io, config)
		}

	}
	// store the first coordinator to ask
	coordinator := config.ID
	if config.DelegateReplication > 0 {
		coordinator += 1
	}

	if config.BatchInterval == 0 {
		glog.Info("Ready to handle requests. No batching enabled")
		// handle client requests (1 at a time)
		for {

			// wait for request
			req := <-io.IncomingRequests
			glog.Info("Request received: ", req)

			// if possible, handle request without replication
			if !req.Replicate {
				io.OutgoingRequests <- req
				glog.Info("Read-only request was handled without replication: ", req)
			} else {
				index++
				glog.Info("Request assigned index: ", index)
				// ok := RunCoordinator(view, index, []msgs.ClientRequest{req}, io, config, true)
				entry := msgs.Entry{view, false, []msgs.ClientRequest{req}}
				coord := msgs.CoordinateRequest{config.ID, view, index, true, entry}
				io.OutgoingUnicast[coordinator].Requests.Coordinate <- coord
				// TODO: BUG: need to handle coordinator failure
				go func() {
					reply := <-(*io).Incoming.Responses.Coordinate
					// TODO: check msg replies to the msg we just sent
					if !reply.Response.Success {
						glog.Warning("Commit unsuccessful")
					}
					glog.Info("Finished replicating request: ", req)
				}()

				// rotate coordinator is nessacary
				if config.DelegateReplication > 1 {
					coordinator += 1
					if coordinator>config.ID + config.DelegateReplication {
						coordinator = config.ID + 1
					}
				}
			}

		}
	} else {
		glog.Info("Ready to handle request. Batch every ", config.BatchInterval, " milliseconds")
		for {
			// setup for holding requests
			reqs := make([]msgs.ClientRequest, config.MaxBatch)
			reqs_num := 0

			// start collecting requests
			timeout := make(chan bool, 1)
			go func() {
				<-time.After(time.Millisecond * time.Duration(config.BatchInterval))
				timeout <- true
			}()

			exit := false
			for exit == false {
				select {
				case req := <-io.IncomingRequests:
					if !req.Replicate {
						io.OutgoingRequests <- req
						glog.Info("Request handled without replication: ", req)
					} else {
						reqs[reqs_num] = req
						glog.Info("Request ", reqs_num, " is : ", req)
						reqs_num = reqs_num + 1
						if reqs_num == config.MaxBatch {
							exit = true
							break
						}
					}
				case <-timeout:
					exit = true
					break
				}
			}

			// assign requests to coordinators
			if reqs_num > 0 {
				glog.Info("Starting to replicate ", reqs_num, " requests")
				reqs_small := reqs[:reqs_num]
				index++
				// ok := RunCoordinator(view, index, reqs_small, io, config, true)
				entry := msgs.Entry{view, false, reqs_small}
				coord := msgs.CoordinateRequest{config.ID, view, index, true, entry}
				io.OutgoingUnicast[coordinator].Requests.Coordinate <- coord
				go func() {
					reply := <- io.Incoming.Responses.Coordinate
					// TODO: check msg replies to the msg we just sent
					if !reply.Response.Success {
						glog.Warning("Commit unsuccessful")
					}
					glog.Info("Finished replicating requests: ", reqs_small)
				} ()
				// rotate coordinator is nessacary
				if config.DelegateReplication > 1 {
					coordinator += 1
					if coordinator>config.ID + config.DelegateReplication {
						coordinator = config.ID + 1
					}
				}
			}
		}

	}

	glog.Warning("Master stepping down")

}
<|endoftext|>


<commit_before>package consumer

import (
	"encoding/json"

	"github.com/bitly/go-nsq"
)

// Message - Inherent nsq
type Message struct {
	*nsq.Message
}

// GiveUp - Finish message with success state because message never will be possible to process
func (m *Message) GiveUp() {
	m.Finish(true)
}

// Success - Finish message as successfully proccessed
func (m *Message) Success() {
	m.Finish(true)
}

// Fail - Mark message as failed to process
func (m *Message) Fail() {
	m.Finish(false)
}

// Finish - Finish processing message
func (m *Message) Finish(success bool) {
	if success {
		m.Message.Finish()
	} else {
		m.Message.Requeue(-1)
	}
}

// ReadJSON - Unmarshals JSON message body to interface.
func (m *Message) ReadJSON(v interface{}) error {
	return json.Unmarshal(m.Body, v)
}
<commit_msg>consumer message context helpers<commit_after>package consumer

import (
	"encoding/json"

	"github.com/bitly/go-nsq"
	"golang.org/x/net/context"
)

// Message - Inherent nsq
type Message struct {
	*nsq.Message
}

var msgkey = "nsqmsg"

// WithMessage - Returns nsq message from context.
func WithMessage(ctx context.Context, msg *Message) context.Context {
	return context.WithValue(ctx, msgkey, msg)
}

// MessageFromContext - Returns nsq message from context.
func MessageFromContext(ctx context.Context) (*Message, bool) {
	value, ok := ctx.Value(msgkey).(*Message)
	return value, ok
}

// GiveUp - Finish message with success state because message never will be possible to process
func (m *Message) GiveUp() {
	m.Finish(true)
}

// Success - Finish message as successfully proccessed
func (m *Message) Success() {
	m.Finish(true)
}

// Fail - Mark message as failed to process
func (m *Message) Fail() {
	m.Finish(false)
}

// Finish - Finish processing message
func (m *Message) Finish(success bool) {
	if success {
		m.Message.Finish()
	} else {
		m.Message.Requeue(-1)
	}
}

// ReadJSON - Unmarshals JSON message body to interface.
func (m *Message) ReadJSON(v interface{}) error {
	return json.Unmarshal(m.Body, v)
}
<|endoftext|>