_This post is part 2 of a 4-part series on monitoring Hadoop health and performance. [Part 1] gives a general overview of Hadoop's architecture and subcomponents, this post covers Hadoop's key metrics, [Part 3] details how to monitor Hadoop performance natively, and [Part 4] explains how to monitor a Hadoop deployment with Datadog._

If you've already [read our guide][Part 1] to Hadoop architecture, you know about the components that make up a typical Hadoop cluster. In this post, we'll dive deep into each of the technologies introduced in that guide and explore the key metrics exposed by Hadoop that you should keep your eye on.

## Service-oriented monitoring

From an operations perspective, Hadoop clusters are incredibly resilient in the face of system failures. Hadoop was designed with failure in mind and can tolerate entire racks going down.

Monitoring Hadoop requires a different mindset than monitoring something like [RabbitMQ]—DataNodes and NodeManagers should be treated [like cattle][pets-v-cattle]. As with Docker containers, you [generally don't care][host-centric monitoring] if a single worker node fails, although failure of a master node such as one of your NameNodes will require fairly rapid remediation. Generally speaking, individual host-level indicators are less important than service-level metrics when it comes to Hadoop.

## Key Hadoop performance metrics to monitor
When working properly, a Hadoop cluster can handle a truly massive amount of data—there are plenty of production clusters managing petabytes of data each. Monitoring each of Hadoop's sub-components is essential to keep jobs running and the cluster humming.

Hadoop metrics can be broken down into four broad categories:

- [HDFS metrics](#toc-hdfs-metrics2)
- [MapReduce counters](#toc-mapreduce-counters4)
- [YARN metrics](#toc-yarn-metrics5)
- [ZooKeeper metrics](#toc-zookeeper-metrics9)

[![Hadoop metrics breakdown diagram][hadoop-metrics-tree]][hadoop-metrics-tree]

To learn more about collecting Hadoop and ZooKeeper metrics, take a look at [part three][Part 3] of this series.

This article references metric terminology introduced in our [Monitoring 101 series][monitoring-101], which provides a framework for metric collection and alerting.

## HDFS metrics
HDFS metrics can be further decomposed into two categories:

- [NameNode metrics](#NameNode-metrics)
- [DataNode metrics](#DataNode-metrics)

<canvas id="canvas" width="700" height="300" style="background-color:#FFFFFF"></canvas>

**NameNode and DataNodes**

[DataNodes][dn-part1] communicate with their [NameNode][nn-part1] (and vice versa) via a periodic heartbeat (by default every three seconds, set by `dfs.heartbeat.interval`). The heartbeat contains (amongst other things) health information about the DataNode. It is through this mechanism that the NameNode is able to know the state of all the DataNodes in the cluster.

Failure of the [NameNode][nn-part1] without any replica standing by will cause data stored in the cluster to become unavailable. Therefore monitoring the primary NameNode and its Secondary/Standby NameNode is critical to ensure high cluster availability. In addition to reporting metrics about themselves, NameNodes can also report metrics about all DataNodes, providing a good birds-eye view of the rest of your HDFS deploy.

NameNode metrics can be divided into two groups:

- NameNode-emitted metrics
- [NameNode JVM metrics](#NameNode-JVM)

### NameNode-emitted metrics

|Name|Description|[Metric Type][monitoring-101]|
|:--:|:--:|:--:|:--:|
|CapacityRemaining|Available capacity|Resource: Utilization|
|CorruptBlocks/MissingBlocks| Number of corrupt/missing blocks|Resource: Error/Resource: Availability|
|VolumeFailuresTotal|Number of failed volumes| Resource: Error|
|NumLiveDataNodes/NumDeadDataNodes|Count of alive DataNodes/Count of dead DataNodes| Resource: Availability|
|FilesTotal|Total count of files tracked by the NameNode| Resource: Utilization|
|TotalLoad|Measure of file access across all DataNodes|Resource: Utilization|
|BlockCapacity/BlocksTotal|Maximum number of blocks allocable/Count of blocks tracked by NameNode| Resource: Utilization
|UnderReplicatedBlocks| Count of under-replicated blocks| Resource: Availability|
|NumStaleDataNodes|Count of stale DataNodes| Resource: Availability|

![HDFS Capacity remaining graph][hdfs-capacity-remaining]

**Metric to alert on: CapacityRemaining**
CapacityRemaining represents the total available capacity remaining across the entire HDFS cluster. It is an understatement to say that running out of disk space in HDFS will cause BadThingsToHappen™. DataNodes that are out of space are likely to fail on boot, and of course, running DataNodes will not be able to be written to. In addition, any running jobs which write out temporary data may fail due to lack of capacity. It is good practice to ensure that disk use never exceeds 80 percent capacity.

Thankfully, HDFS [makes it easy][hdfs-add-datanode] to spin up new DataNodes to add capacity (or you can add extra disks to existing DataNodes, of course).

**Metric to alert on: MissingBlocks**
Corrupt or missing blocks can point to an unhealthy cluster. Typically, upon receiving a read request a DataNode will locate the requested block and serve it back to the client, along with a checksum for the data (which was calculated and stored on the DataNode when the block was written). If the checksum for the received data doesn’t match the stored checksum, the client reports the corruption to the NameNode and reads the data from one of the other replicas. The NameNode, meanwhile, schedules a rereplication of the block from one of the healthy copies, and then tells the original DataNode to delete its corrupted copy. So, although you may see corrupt blocks reported on your cluster from time to time, that doesn’t necessarily mean that you have lost any data.

A _missing_ block is far worse than a corrupt block, because a missing block cannot be recovered by [copying a replica][hadoop-recovery]. A missing block represents a block for which no known copy exists in the cluster. That's not to say that the block no longer exists—if a series of DataNodes were taken offline for maintenance, missing blocks could be reported until they are brought back up.

You may not care if you have a corrupt block or two in the entire cluster, but if blocks start going missing without any scheduled maintenance it is cause for concern.

**Metric to alert on: VolumeFailuresTotal**
In [old versions of Hadoop][datanode-fail], failure of a single disk would cause an entire DataNode to be taken offline. Thankfully, that is no longer the case—HDFS now allows for disks to fail in place, without affecting DataNode operations, until a threshold value is reached. (This is set on each DataNode via the `dfs.datanode.failed.volumes.tolerated` property; it defaults to 0, meaning that any volume failure will shut down the DataNode; on a production cluster where DataNodes typically have 6, 8, or 12 disks, setting this parameter to 1 or 2 is typically the best practice.)

Though a failed volume will not bring your cluster to grinding halt, you most likely want to know when hardware failures occur, if only so that you can replace the failed hardware. Since HDFS was designed with hardware failure in mind, with a default replication of three, a failed volume should never signal data loss.

**Metric to alert on: NumDeadDataNodes**
NumLiveDataNodes and NumDeadDataNodes together track the number of DataNodes in the cluster, by state. Ideally, your number of live DataNodes will be equal to the number of DataNodes you've provisioned for the cluster. If this number unexpectedly drops, it may warrant investigation.

NumDeadDataNodes, on the other hand, should be alerted on. When the NameNode does not hear from a DataNode for 30 seconds, that DataNode is marked as "stale." Should the DataNode fail to communicate with the NameNode for 10 minutes following the transition to the "stale" state, the DataNode is marked "dead." The death of a DataNode causes a flurry of network activity, as the NameNode initiates replication of blocks lost on the dead nodes.

Though the loss of a single DataNode may not impact performance much, losing multiple DataNodes will start to be very taxing on cluster resources, and could result in data loss.

**FilesTotal** is a running count of the number of files being tracked by the NameNode. Because the NameNode stores all metadata in memory, the greater the number of files (or blocks, or directories) tracked, [the more memory][small-files-problem] required by NameNode. The rule of thumb for NameNode memory requirements is that each object (file, directory, block) tracked by the NameNode consumes roughly [150 bytes][namenode-limit] of memory. If each file is mapped to one block (files often span multiple blocks), that is approximately 300 bytes per file. That doesn't sound like much until you factor in the replication of data: the default replication factor is 3 (so 900 bytes per file), _and_ each replica adds an additional 16 bytes, for a total of ~1 KB of metadata per file. That still may not sound like much, but as your cluster expands to include millions of files, you should keep this number in mind and plan to add additional memory to your NameNode.

![TotalLoad diagram][hdfs-totalload]

**TotalLoad** is the current number of concurrent file accesses (read/write) across all DataNodes. This metric is not usually indicative of a problem, though it can illuminate issues related to job execution. Because worker nodes running the DataNode daemon also perform MapReduce tasks, extended periods of high I/O (indicated by high TotalLoad) generally translate to degraded job execution performance. Tracking TotalLoad over time can help get to the bottom of job performance issues.

**BlockCapacity/BlocksTotal**: Like FilesTotal, keeping an eye on the total number of blocks across the cluster is essential to continued operation. Block capacity and file capacity are very closely related but not the same (files can span multiple blocks); monitoring both block and file capacity is necessary to ensure uninterrupted operation. If your cluster is approaching the limits of block capacity, you can easily add more by bringing up a new DataNode and adding it to the pool, or adding more disks to existing nodes. Once added, you can use the [Hadoop balancer][balancer] to balance the distribution of blocks across DataNodes.

**UnderReplicatedBlocks** are the number of blocks with insufficient replication. Hadoop's replication factor is configurable on a [per-client][global-replication] or [per-file][file-replication] basis. The default replication factor is three, meaning that each block will be stored on three DataNodes. If you see a large, sudden spike in the number of under-replicated blocks, it is likely that a DataNode has died—this can be verified by correlating under-replicated block metrics with the status of DataNodes (via NumLiveDataNodes and related metrics).

[![Live, dead, and decommissioned DataNodes image][live-dead-decom]][live-dead-decom]

**NumStaleDataNodes**: Every three seconds (by default), DataNodes send a heartbeat message to the NameNode. This heartbeat acts as a health-check mechanism but can also be used by the NameNode to delegate new work. As explained above, DataNodes that do not send a heartbeat within 30 seconds are marked as "stale." A DataNode might not emit a heartbeat for a number of reasons—network issues or high load are among the more common causes. Unless you've disabled reads or writes to stale DataNodes, you should definitely keep an eye on this metric. A DataNode that did not emit a heartbeat due to high system load (or a slow network) will likely cause any reads or writes to that DataNode to be delayed as well. However, in practice, a stale DataNode is likely dead.

Even if reading and writing on stale DataNodes is disabled, you should still monitor for sudden increases in the number of stale DataNodes, because in a short time stale nodes can easily become dead nodes. And stale DataNodes will be written to regardless of your settings if the ratio of stale DataNodes to total DataNodes exceeds the value of `dfs.namenode.write.stale.datanode.ratio`.

### NameNode JVM metrics
Because the NameNode runs in the Java Virtual Machine (JVM), it relies on Java garbage collection processes to free up memory. The more activity in your HDFS cluster, the more often the garbage collection will run.

Anyone familiar with Java applications knows that garbage collection can come with a high performance cost. See [the Oracle documentation][jvm-gc] for a good primer on Java garbage collection.

If you are seeing excessive pauses during garbage collection, you can consider upgrading your JDK version or garbage collector. Additionally, you can tune your Java runtime to minimize garbage collection.

Because a NameNode is typically hosted on a machine with ample computational resources, using the CMS (ConcurrentMarkSweep) garbage collector is recommended, _especially_ because bottlenecks on the NameNode will result in degraded performance cluster-wide. From the Oracle documentation:
> The CMS collector should be used for applications that require low pause times and can share resources with the garbage collector. Examples include desktop UI application that respond to events, a webserver responding to a request or a database responding to queries.

ConcurrentMarkSweep collections free up unused memory in the old generation of the heap. CMS is a low-pause garbage collection, meaning that although it does temporarily stop application threads, it does so only intermittently.

For more information on optimizing NameNode garbage collection, see this [excellent guide][hadoop-jvm-tuning] from Hortonworks.

|Name|Description|[Metric Type][monitoring-101]|
|:--:|:--:|:--:|:--:|
|ConcurrentMarkSweep count|Number of old-generation collections|Other |
|ConcurrentMarkSweep time| Elapsed time of old-generation collections, in milliseconds| Other|

[![NameNode CMS][nn-cms-graph]][nn-cms-graph]

If CMS is taking more than a few seconds to complete, or is occurring with increased frequency, your NameNode may not have enough memory allocated to the JVM to function efficiently.

### DataNode metrics

DataNode metrics are host-level metrics specific to a particular DataNode.

|Name|Description|[Metric Type][monitoring-101]|
|:--:|:--:|:--:|:--:|
|Remaining| Remaining disk space | Resource: Utilization|
|NumFailedVolumes| Number of failed storage volumes | Resource: Error|

[![Disk remaining on DataNodes][dn-disk-graph]][dn-disk-graph]

**Remaining**: As mentioned in the [section on NameNode metrics](#Capacity), running out of disk space can cause a number of problems in a cluster. If left unrectified, a single DataNode running out of space could quickly cascade into failures across the entire cluster as data is written to an increasingly-shrinking pool of available DataNodes. Tracking this metric over time is essential to maintain a healthy cluster; you may want to alert on this metric when the remaining space falls dangerously low (less than 10 percent).

**NumFailedVolumes**: By default, a single volume failing on a DataNode will cause the entire node to go offline. Depending on your environment, that may not be desired, especially since most DataNodes operate with multiple disks. If you haven't set `dfs.datanode.failed.volumes.tolerated` in your HDFS configuration, you should definitely track this metric. Remember, when a DataNode is taken offline, the NameNode must copy any under-replicated blocks that were lost on that node, causing a burst in network traffic and potential performance degradation. To prevent a slew of network activity from the failure of a single disk (if your use case permits), you should set the tolerated level of volume failure in your config like so:

```
<property>
    <name>dfs.datanode.failed.volumes.tolerated</name>
    <value>N</value>
</property>
```
where N is the number of failures a DataNode should tolerate before shutting down.
_Don't forget to restart your DataNode process following a configuration change._

The downside to enabling this feature is that, with fewer disks to write to, a DataNode's performance may suffer. Setting a reasonable value for tolerated failures will vary depending on your use case.

## MapReduce counters

The MapReduce framework exposes a number of counters to track statistics on MapReduce job execution. Counters are an invaluable mechanism that let you see what is actually happening during a MapReduce job run. They provide a count of the number of times an event occurred. MapReduce counters can be broken down into four categories:

- [job counters](#Job-counters)
- [task counters](#Task-counters)
- [custom counters](#Custom-counters)
- file system counters

[![MapReduce counters breakdown][mapreduce-metrics-tree]][mapreduce-metrics-tree]

MapReduce application developers may want to track the following counters to to find bottlenecks and potential optimizations in their applications.

### Job counters
Group name:`org.apache.hadoop.mapreduce.JobCounter`

|Counter Name|Description|[Metric Type][monitoring-101]|
|:---:|:---:|:---:|
|MILLIS\_MAPS/MILLIS\_REDUCES| Processing time for maps/reduces|Work: Performance|
|NUM\_FAILED\_MAPS/NUM\_FAILED\_REDUCES| Number of failed maps/reduces|Work: Error|
|RACK\_LOCAL\_MAPS/DATA\_LOCAL\_MAPS/OTHER\_LOCAL\_MAPS| Counters tracking where map tasks were executed| Resource: Other|

**MILLIS\_MAPS/MILLIS\_REDUCES**: These metrics track the wall time spent across all of your map and reduce tasks. Their full utility is realized when tracked alongside the task counter GC\_TIME\_MILLIS (see [below](#toc-task-counters8)); when taken altogether, it is possible to determine the percentage of time spent performing garbage collection.

Using all three counters, we can calculate the percentage of time spent in garbage collection with the following formula: GC\_TIME\_PERCENTAGE = GC\_TIME\_MILLIS / (MILLIS\_MAPS + MILLIS\_REDUCES)

Jobs where garbage collection is consuming a significant percentage of computation time (e.g. more than 20 percent) may benefit from increasing the amount of heap available to them.

**NUM\_FAILED\_MAPS/NUM\_FAILED\_REDUCES**: This pair of metrics tracks the number of failed map/reduce tasks for a job. Tasks can fail for many reasons—abrupt exit by the JVM, runtime errors, and hanging tasks are among the most common. Failed tasks are tolerated, up to the value set by the `mapreduce.map.maxattempts` and `mapreduce.reduce.maxattempts` properties (set to four, by default). If a task fails `maxattempts` times, it is marked as failed. A job will fail as a whole if more than `mapreduce.map.failures.maxpercent` percent of map tasks in the job fail or more than `mapreduce.reduce.failures.maxpercent` percent of the reduce tasks fail.

Because failures are expected, you should not alert on this metric, though if you are seeing the same task fail on several DataNodes, it may be worth diving into the job logs or even the application code for bugs. If many tasks fail on a particular DataNode, the issue could be hardware-related.

**DATA\_LOCAL\_MAPS/RACK\_LOCAL\_MAPS/OTHER\_LOCAL\_MAPS**: These counters track the number of map tasks that were performed, aggregated by location. Data can be located either: on the node executing the map task, on a node in the same rack as the node performing the map task, or on a node located on a different rack somewhere else in the cluster. Recall from [part 1 of this series][Part-1-HDFS] that one of Hadoop's greatest strengths lies in its preference for moving computation as close to the data as possible. The benefits from operating on local data include reduced network overhead and faster execution. These benefits are lost when the computation must be performed on a node that is not physically close to the data being processed.

Because sending data over the network takes time, correlating poor job performance with these counters can help determine _why_ performance suffered. If many maps had to be executed on nodes where the data was not locally hosted, it would be reasonable to expect degraded performance.

### Task counters
Task counters track the results of task operations in aggregate. Each counter below represents the total across all tasks associated with a particular job.

Group name: `org.apache.hadoop.mapreduce.JobCounter`

|Counter Name|Description|[Metric Type][monitoring-101]|
|:---:|:---:|:---:|
|REDUCE\_INPUT\_RECORDS| Number of input records for reduce tasks|Other|
|SPILLED\_RECORDS|Number of records spilled to disk|Resource: Saturation|
|GC\_TIME\_MILLIS|Processing time spent in garbage collection|Other|

**REDUCE\_INPUT\_RECORDS**: Keeping an eye on the number of input records for each reduce task in a job is one of the best ways to identify performance issues related to a _skewed dataset_. Skewed data is characterized by a disproportionate distribution of input records across all reducers for a job. To give a concrete example, consider the canonical wordcount run on a text composed of 40,000 "the"s and one "a". If this text were run through two reducers, and each got the values associated with one of the two keys, the reducer working on the larger dataset (the set of "the" keys) would take much longer to complete than the reducer with a much smaller dataset.

Skewed datasets can make jobs take much longer to complete than expected. Unfortunately, cluster administrators can't simply add more reducers to speed up execution—in cases like this, the MapReduce job would probably benefit from the addition of a [combine step][combiner].

**SPILLED\_RECORDS**: Map output data is stored in memory, but when the buffer fills up data is dumped to disk. A separate thread is spawned to merge all of the spilled data into a single larger sorted file for reducers. Spills to disk should be minimized, as more than one spill will result in an increase in disk activity as the merging thread has to read and write data to disk. Tracking this metric can help determine if configuration changes need to be made (like increasing the memory available for map tasks) or if additional processing (compression) of the input data is necessary.

**GC\_TIME\_MILLIS**: Useful for determining the percentage of time tasks spent performing garbage collection. See the section on [MILLIS\_MAPS/MILLIS\_REDUCES](#MILLIS-MAPS) for more information.

### Custom counters
MapReduce allows users to implement [custom counters][counter-api] that are specific to their application. Custom counters can be used to track more fine-grained counts, such as [counting the number of malformed or missing records][custom-counter-tut].

Keep in mind, however, that since all counters reside in the JobTracker JVM's memory, there is a practical limit to the number of counters you should use, limited by the amount of physical memory available.  (Hadoop has a configuration value, `mapreduce.job.counters.max`, which limits the number of custom counters that can be created; by default this is 120.)

## YARN metrics
YARN metrics can be grouped into three distinct categories:

- [Cluster metrics](#toc-cluster-metrics6)
- [Application metrics](#toc-application-metrics7)
- [NodeManager metrics](#toc-nodemanager-metrics8)

[![YARN metrics breakdown][yarn-metrics-tree]][yarn-metrics-tree]

### Cluster metrics
Cluster metrics provide a high-level view of YARN application execution. The metrics listed here are aggregated cluster-wide.

|Name|Description|[Metric Type][monitoring-101]|
|:--:|:--:|:--:|
|unhealthyNodes|Number of unhealthy nodes|Resource: Error|
|activeNodes|Number of currently active nodes|Resource: Availability|
|lostNodes|Number of lost nodes|Resource: Error|
|appsFailed|Number of failed applications|Work: Error|
|totalMB/allocatedMB|Total amount of memory/amount of memory allocated|Resource: Utilization|

**Metric to alert on: unhealthyNodes**
YARN considers any node with disk utilization exceeding the value specified under the property `yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage` (in yarn-site.xml) to be unhealthy. Ample disk space is critical to ensure uninterrupted operation of a Hadoop cluster, and large numbers of unhealthyNodes (the number to alert on depends on the size of your cluster) should be quickly investigated and resolved. Once a node is marked unhealthy, other nodes have to fill the void, which can cause a cascade of alerts as node after node nears full disk utilization.

It is worth mentioning that you can add your own health checks to YARN with a [simple configuration change][external-health-script]. But be warned—if the script should fail for any reason, including wrong permissions, incorrect path, and so on—it will cause the node to be marked as unhealthy.

**activeNodes/lostNodes**: The activeNodes metric tracks the current count of active, or normally operating, nodes. This metric should remain static in the absence of anticipated maintenance. If a NodeManager fails to maintain contact with the ResourceManager, it will eventually be marked as "lost" and its resources will become unavailable for allocation. The timeout threshold is configurable via the `yarn.resourcemanager.nm.liveness-monitor.expire-interval-ms` property in yarn-site.xml, with a default value of 10 minutes (600000 ms). Nodes can lose contact with the ResourceManager for a number of reasons, including network issues, hardware failure, or a hung process. It is good to be aware of lost nodes, so you can take action if the node never returns.

**appsFailed**: Assuming you are running MapReduce on YARN, if the percentage of failed map or reduce tasks exceeds a specific threshold (configured with `mapreduce.map.failures.maxpercent` and `mapreduce.reduce.failures.maxpercent`, respectively) the application as a whole will fail. It is important to note that YARN cannot distinguish between a failed NodeManager and a hanging task—in either case, the NodeManager will fail to report to the ResourceManager within the heartbeat timeout period and the task will be rerun. After four unsuccessful attempts to run the task (configured with `mapreduce.map.maxattempts` and `mapreduce.reduce.maxattempts`, respectively), it will be marked as failed.

In the case of ResourceManager failure, application state is largely unaffected, though no new work will be able to be submitted and currently running jobs will start to fail. Because the ApplicationMasters (on each NodeManager) store the application state (including the list of run/failed/killed tasks), once a new ResourceManager is brought online (and assuming the old saved state is recoverable), NodeManagers can pick up where they left off.

**totalMB/allocatedMB**: When taken together, the totalMB and allocatedMB metrics give a high-level view of your cluster's memory usage. Memory is currency in Hadoop, and if you are nearing the ceiling of your memory usage you have a couple of options. To increase the memory available to your cluster, you can add new NodeManager nodes, tweak the amount of memory reserved for YARN applications (set by the `yarn.nodemanager.resource.memory-mb` in `yarn-site.xml`, or change the minimum amount of RAM allocated to containers (set by the `yarn.scheduler.minimum-allocation-mb` property in `yarn-site.xml`).

Keep in mind that YARN may over-commit resources, which can occasionally translate to reported values of allocatedMB which are higher than totalMB.

### Application metrics
Application metrics provide detailed information on the execution of individual YARN applications.

|Name|Description|[Metric Type][monitoring-101]|
|:--:|:--:|:--:|
|progress|Application execution progress meter|Work: Performance|

**progress**: Progress gives you a real-time window into the execution of a YARN application. Its reported value will always be in the range of zero to one (inclusive), with a value of one indicating completed execution. Because application execution can often times be opaque when running hundreds of applications on thousands of nodes, tracking progress alongside other metrics can better help you to determine the cause of any performance degradation. Applications that go extended periods without making progress should be investigated.

### NodeManager metrics
NodeManager metrics provide resource information at the individual node level.

|Name|Description|[Metric Type][monitoring-101]|
|:--:|:--:|:--:|
|containersFailed| Number of containers that failed to launch | Resource: Error|

**containersFailed** tracks the number of containers that failed to launch on that particular NodeManager. [Recall from the first article][resource-containers] in the series that NodeManager containers are _not_ Docker containers.
[Configuration issues aside][container-launch-fail], the most common cause of container launch failure is hardware-related. If the NodeManager's disk is full, [any container it launches will fail][container-fail-on-disk]. Similarly, if the NodeManager's [heap is set too low][container-heap], containers won't launch. You don't need to be alerted for every container that failed to launch, but if a single node has many failed containers, it may be time to investigate.

## ZooKeeper metrics
ZooKeeper plays an important role in a Hadoop deployment. It is responsible for ensuring the availability of the [HDFS NameNode][hdfs-zookeeper] and YARN's ResourceManager. If your cluster is operating in high-availability mode, you should be sure to monitor ZooKeeper alongside the rest of your Hadoop components to avoid service interruptions.

ZooKeeper exposes metrics via MBeans as well as through a command line interface, using the so-called [4-letter words][part3-4-letters].  For more details on collecting ZooKeeper metrics, be sure to check out [part three][Part 3] of this series.

|Name|Description|[Metric Type][monitoring-101]|
|:--:|:--:|:--:|
|`zk_followers`|Number of active followers| Resource: Availability|
|`zk_avg_latency`|Amount of time it takes to respond to a client request (in ms)| Work: Performance|
|`zk_num_alive_connections`| Number of clients connected to ZooKeeper | Resource: Availability|

**Metric to alert on: zk\_followers (leader only)**
The number of followers should equal the total size of your ZooKeeper ensemble, minus 1 (the leader is not included in the follower count). If the ensemble fails to maintain quorum, all automatic failover features are suspended. Changes to this value should be alerted on, as the size of your ensemble should only change due to user intervention (e.g., an administrator decommissioned or commissioned a node).

**zk\_avg\_latency**: The average request latency is the average time it takes (in milliseconds) for ZooKeeper to respond to a request. ZooKeeper will not respond to a request until it has written the transaction to its transaction log. Any latency spikes could delay the transition between servers in the event of a ResourceManager or NameNode failure. Of course, such events are rare, but by tracking the average latency over time you can better correlate errors that may have been caused by stale data.

[![ZooKeeper average request latency][avg-req-latency-zk]][avg-req-latency-zk]

**zk\_num\_alive\_connections**: ZooKeeper reports the number of connected clients via this metric. This represents all connections, including connections from non-ZooKeeper nodes. In most environments, this number should remain fairly static—generally, your number of NameNodes, ResourceManagers, and their respective standbys should remain relatively stable. You should be aware of unanticipated drops in this value; since Hadoop uses ZooKeeper to provide high availability, a loss of connection to ZooKeeper could make the cluster less resilient to node loss.

## Go forth, and collect!
In this post we’ve explored many of the key metrics you should monitor to keep tabs on the health and performance of your Hadoop cluster.

The metrics covered in this post pertain to the core Hadoop components and their typical use. Given Hadoop's vast ecosystem, over time you will likely identify additional metrics that are particularly relevant to your specific Hadoop infrastructure, and its users.

[Read on][Part 3] for a comprehensive guide to collecting all of the metrics described in this article, and any other metric exposed by Hadoop.

## Acknowledgments

Special thanks to [Ian Wrigley][ian-twit], Director of Education Services at [Confluent][confluent], for graciously sharing both his Hadoop expertise and monitoring strategies for this article.

[ian-twit]: https://twitter.com/iwrigley
[confluent]: http://www.confluent.io/

_Source Markdown for this post is available [on GitHub][markdown]. Questions, corrections, additions, etc.? Please [let us know][issues]._

[]: Links

[]: Preamble

[host-centric monitoring]: https://www.datadoghq.com/blog/the-docker-monitoring-problem/#toc-host-centric-monitoring14
[monitoring-101]: https://www.datadoghq.com/blog/monitoring-101-collecting-data/
[pets-v-cattle]: http://cloudscaling.com/blog/cloud-computing/pets-vs-cattle-the-elastic-cloud-story/
[RabbitMQ]: https://www.datadoghq.com/blog/openstack-monitoring-nova/#rabbitmq-metrics
[zookeeper-ha]: REPLACE_LINK_TO_ZOOKEEPER_IN_PART0

[]: HDFS

[balancer]: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#balancer
[datanode-fail]: https://issues.apache.org/jira/browse/HDFS-2137
[decommission-datanode]: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Sys_Admin_Guides/content/ref-a179736c-eb7c-4dda-b3b4-6f3a778bd8c8.1.html
[file-replication]: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html#setrep
[global-replication]: https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml#dfs.replication
[hadoop-jvm-tuning]: https://community.hortonworks.com/articles/14170/namenode-garbage-collection-configuration-best-pra.html
[hadoop-recovery]: https://twiki.opensciencegrid.org/bin/view/Storage/HadoopRecovery
[hdfs-add-datanode]: https://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
[namenode-limit]: https://www.mail-archive.com/core-user@hadoop.apache.org/msg02835.html
[small-files-problem]: http://blog.cloudera.com/blog/2009/02/the-small-files-problem/
[jvm-gc]: http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html

[]: MapReduce

[combiner]: http://www.tutorialspoint.com/map_reduce/map_reduce_combiners.htm
[counter-api]: https://hadoop.apache.org/docs/r2.7.2/api/org/apache/hadoop/mapred/Counters.html
[custom-counter-tut]: http://www.ashishpaliwal.com/blog/2012/05/hadoop-recipe-using-custom-java-counters/

[]: YARN

[container-launch-fail]: https://stackoverflow.com/questions/22579943/yarn-mapreduce-job-issue-am-container-launch-error-in-hadoop-2-3-0
[container-fail-on-disk]: https://issues.apache.org/jira/browse/YARN-257
[container-heap]: https://stackoverflow.com/questions/23374693/hadoop-yarn-failed-to-launch-container
[external-health-script]: https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/NodeManager.html#External_Health_Script

[]: ZooKeeper

[avg-req-latency-zk]: https://don08600y3gfm.cloudfront.net/ps3b/blog/images/2016-07-hadoop/graphs/zk-lat.png

[hdfs-zookeeper]: https://www.datadoghq.com/blog/hadoop-architecture-overview/#toc-hdfs-and-zookeeper16
[yarn-zookeeper]: https://www.datadoghq.com/blog/hadoop-architecture-overview/#toc-hdfs-and-zookeeper16

[]: Images

[hdfs-capacity-remaining]: https://don08600y3gfm.cloudfront.net/ps3b/blog/images/2016-07-hadoop/graphs/hdfs-disk-usage-2.png
[hdfs-totalload]: https://don08600y3gfm.cloudfront.net/ps3b/blog/images/2016-07-hadoop/graphs/hdfs-totalload.png
[dn-disk-graph]: https://don08600y3gfm.cloudfront.net/ps3b/blog/images/2016-07-hadoop/graphs/dn-disk0.png
[live-dead-decom]: https://don08600y3gfm.cloudfront.net/ps3b/blog/images/2016-07-hadoop/graphs/live-dead-decom.png
[nn-cms-graph]: https://don08600y3gfm.cloudfront.net/ps3b/blog/images/2016-07-hadoop/graphs/nn-cms-1.png

[hadoop-metrics-tree]: https://don08600y3gfm.cloudfront.net/ps3b/blog/images/2016-07-hadoop/metrics-diagrams/hadoop_diagram10-5.png
[hdfs-metrics-tree]: https://don08600y3gfm.cloudfront.net/ps3b/blog/images/2016-07-hadoop/metrics-diagrams/hadoop_diagram11-1.png
[mapreduce-metrics-tree]: https://don08600y3gfm.cloudfront.net/ps3b/blog/images/2016-07-hadoop/metrics-diagrams/hadoop_diagram12-2.png
[yarn-metrics-tree]: https://don08600y3gfm.cloudfront.net/ps3b/blog/images/2016-07-hadoop/metrics-diagrams/hadoop_diagram13-3.png

[Part 1]: https://www.datadoghq.com/blog/hadoop-architecture-overview/
[Part 2]: https://www.datadoghq.com/blog/monitor-hadoop-metrics/
[Part 3]: https://www.datadoghq.com/blog/collecting-hadoop-metrics/
[Part 4]: https://www.datadoghq.com/blog/monitor-hadoop-metrics-datadog/
[Part-1-HDFS]: https://www.datadoghq.com/blog/hadoop-architecture-overview/#HDFS-architecture
[resource-containers]: https://www.datadoghq.com/blog/hadoop-architecture-overview/#Glossary
[part3-4-letters]: https://www.datadoghq.com/blog/collecting-hadoop-metrics/#4-letter-words

[dn-part1]: https://www.datadoghq.com/blog/hadoop-architecture-overview/#HDFS-architecture
[nn-part1]: https://www.datadoghq.com/blog/hadoop-architecture-overview/#NameNode

[markdown]: https://github.com/DataDog/the-monitor/blob/master/hadoop/how_to_monitor_hadoop_metrics.md
[issues]: https://github.com/DataDog/the-monitor/issues

{'repo_name': 'DataDog/the-monitor', 'stars': '466', 'repo_language': 'Python', 'file_name': 'monitoring-aws-lambda-with-datadog.md', 'mime_type': 'text/plain', 'hash': 566581864313757000, 'source_dataset': 'data'}


/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to you under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.calcite.test;

import org.apache.calcite.avatica.util.Casing;
import org.apache.calcite.avatica.util.Quoting;
import org.apache.calcite.config.Lex;
import org.apache.calcite.rel.type.RelDataType;
import org.apache.calcite.sql.SqlCollation;
import org.apache.calcite.sql.SqlNode;
import org.apache.calcite.sql.SqlOperatorTable;
import org.apache.calcite.sql.parser.SqlParseException;
import org.apache.calcite.sql.test.AbstractSqlTester;
import org.apache.calcite.sql.test.SqlTestFactory;
import org.apache.calcite.sql.test.SqlTester;
import org.apache.calcite.sql.test.SqlValidatorTester;
import org.apache.calcite.sql.validate.SqlConformance;
import org.apache.calcite.sql.validate.SqlConformanceEnum;
import org.apache.calcite.sql.validate.SqlMonotonicity;
import org.apache.calcite.sql.validate.SqlValidator;
import org.apache.calcite.test.catalog.MockCatalogReaderExtended;
import org.apache.calcite.testlib.annotations.WithLex;

import com.google.common.base.Preconditions;

import org.junit.jupiter.api.extension.BeforeEachCallback;
import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.platform.commons.support.AnnotationSupport;

import java.nio.charset.Charset;
import java.util.Objects;
import java.util.function.UnaryOperator;
import javax.annotation.Nonnull;

import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.MatcherAssert.assertThat;

/**
 * An abstract base class for implementing tests against {@link SqlValidator}.
 *
 *
A derived class can refine this test in two ways. First, it can add
 * testXxx() methods, to test more functionality.
 *
 *
Second, it can override the {@link #getTester} method to return a
 * different implementation of the {@link Tester} object. This encapsulates the
 * differences between test environments, for example, which SQL parser or
 * validator to use.

 */
public class SqlValidatorTestCase {
  private static final SqlTestFactory EXTENDED_TEST_FACTORY =
      SqlTestFactory.INSTANCE.withCatalogReader(MockCatalogReaderExtended::new);

  static final SqlTester EXTENDED_CATALOG_TESTER =
      new SqlValidatorTester(EXTENDED_TEST_FACTORY);

  static final SqlTester EXTENDED_CATALOG_TESTER_2003 =
      new SqlValidatorTester(EXTENDED_TEST_FACTORY)
          .withConformance(SqlConformanceEnum.PRAGMATIC_2003);

  static final SqlTester EXTENDED_CATALOG_TESTER_LENIENT =
      new SqlValidatorTester(EXTENDED_TEST_FACTORY)
          .withConformance(SqlConformanceEnum.LENIENT);

  protected SqlTester tester;

  /**
   * Creates a test case.
   */
  public SqlValidatorTestCase() {
    this.tester = getTester();
  }

  //~ Methods ----------------------------------------------------------------

  /**
   * Returns a tester. Derived classes should override this method to run the
   * same set of tests in a different testing environment.
   */
  public SqlTester getTester() {
    return new SqlValidatorTester(SqlTestFactory.INSTANCE);
  }

  /** Creates a test context with a SQL query. */
  public final Sql sql(String sql) {
    return new Sql(tester, sql, true, false);
  }

  /** Creates a test context with a SQL expression. */
  public final Sql expr(String sql) {
    return new Sql(tester, sql, false, false);
  }

  /** Creates a test context with a SQL expression.
   * If an error occurs, the error is expected to span the entire expression. */
  public final Sql wholeExpr(String sql) {
    return expr(sql).withWhole(true);
  }

  public final Sql winSql(String sql) {
    return sql(sql);
  }

  public final Sql win(String sql) {
    return sql("select * from emp " + sql);
  }

  public Sql winExp(String sql) {
    return winSql("select " + sql + " from emp window w as (order by deptno)");
  }

  public Sql winExp2(String sql) {
    return winSql("select " + sql + " from emp");
  }

  /**
   * Encapsulates differences between test environments, for example, which
   * SQL parser or validator to use.
   *
   *
It contains a mock schema with EMP and DEPT
   * tables, which can run without having to start up Farrago.
   */
  public interface Tester {
    SqlNode parseQuery(String sql) throws SqlParseException;

    SqlNode parseAndValidate(SqlValidator validator, String sql);

    SqlValidator getValidator();

    /**
     * Checks that a query is valid, or, if invalid, throws the right
     * message at the right location.
     *
     *
If expectedMsgPattern is null, the query must
     * succeed.
     *
     *
If expectedMsgPattern is not null, the query must
     * fail, and give an error location of (expectedLine, expectedColumn)
     * through (expectedEndLine, expectedEndColumn).
     *
     * @param sql                SQL statement
     * @param expectedMsgPattern If this parameter is null the query must be
     *                           valid for the test to pass; If this parameter
     *                           is not null the query must be malformed and the
     *                           message given must match the pattern
     */
    void assertExceptionIsThrown(
        String sql,
        String expectedMsgPattern);

    /**
     * Returns the data type of the sole column of a SQL query.
     *
     *
For example, getResultType("VALUES (1") returns
     * INTEGER.
     *
     *
Fails if query returns more than one column.
     *
     * @see #getResultType(String)
     */
    RelDataType getColumnType(String sql);

    /**
     * Returns the data type of the row returned by a SQL query.
     *
     *
For example, getResultType("VALUES (1, 'foo')")
     * returns RecordType(INTEGER EXPR$0, CHAR(3) EXPR#1).
     */
    RelDataType getResultType(String sql);

    void checkCollation(
        String sql,
        String expectedCollationName,
        SqlCollation.Coercibility expectedCoercibility);

    void checkCharset(
        String sql,
        Charset expectedCharset);

    /**
     * Checks that a query returns one column of an expected type. For
     * example, checkType("VALUES (1 + 2)", "INTEGER NOT
     * NULL").
     */
    void checkColumnType(
        String sql,
        String expected);

    /**
     * Given a SQL query, returns a list of the origins of each result
     * field.
     *
     * @param sql             SQL query
     * @param fieldOriginList Field origin list, e.g.
     *                        "{(CATALOG.SALES.EMP.EMPNO, null)}"
     */
    void checkFieldOrigin(String sql, String fieldOriginList);

    /**
     * Checks that a query gets rewritten to an expected form.
     *
     * @param query           query to test
     * @param expectedRewrite expected SQL text after rewrite and unparse
     */
    void checkRewrite(String query, String expectedRewrite);

    /**
     * Checks that a query returns one column of an expected type. For
     * example, checkType("select empno, name from emp""{EMPNO INTEGER
     * NOT NULL, NAME VARCHAR(10) NOT NULL}").
     */
    void checkResultType(
        String sql,
        String expected);

    /**
     * Checks if the interval value conversion to milliseconds is valid. For
     * example, checkIntervalConv(VALUES (INTERVAL '1' Minute),
     * "60000").
     */
    void checkIntervalConv(
        String sql,
        String expected);

    /**
     * Given a SQL query, returns the monotonicity of the first item in the
     * SELECT clause.
     *
     * @param sql SQL query
     * @return Monotonicity
     */
    SqlMonotonicity getMonotonicity(String sql);

    SqlConformance getConformance();
  }

  /** Fluent testing API. */
  static class Sql {
    private final SqlTester tester;
    private final String sql;
    private final boolean query;
    private final boolean whole;

    /** Creates a Sql.
     *
     * @param tester Tester
     * @param sql SQL query or expression
     * @param query True if {@code sql} is a query, false if it is an expression
     * @param whole Whether the failure location is the whole query or
     *              expression
     */
    Sql(SqlTester tester, String sql, boolean query, boolean whole) {
      this.tester = tester;
      this.query = query;
      this.sql = sql;
      this.whole = whole;
    }

    Sql withTester(UnaryOperator<SqlTester> transform) {
      return new Sql(transform.apply(tester), sql, query, whole);
    }

    public Sql sql(String sql) {
      return new Sql(tester, sql, true, false);
    }

    public Sql expr(String sql) {
      return new Sql(tester, sql, false, false);
    }

    public String toSql(boolean withCaret) {
      final String sql2 = withCaret
          ? (whole ? ("^" + sql + "^") : sql)
          : (whole ? sql : sql.replace("^", ""));
      return query ? sql2 : AbstractSqlTester.buildQuery(sql2);
    }

    Sql withExtendedCatalog() {
      return withTester(tester -> EXTENDED_CATALOG_TESTER);
    }

    public Sql withQuoting(Quoting quoting) {
      return withTester(tester -> tester.withQuoting(quoting));
    }

    Sql withLex(Lex lex) {
      return withTester(tester -> tester.withLex(lex));
    }

    Sql withConformance(SqlConformance conformance) {
      return withTester(tester -> tester.withConformance(conformance));
    }

    Sql withTypeCoercion(boolean typeCoercion) {
      return withTester(tester -> tester.enableTypeCoercion(typeCoercion));
    }

    Sql withWhole(boolean whole) {
      Preconditions.checkArgument(sql.indexOf('^') < 0);
      return new Sql(tester, sql, query, whole);
    }

    Sql ok() {
      tester.assertExceptionIsThrown(toSql(false), null);
      return this;
    }

    /**
     * Checks that a SQL expression gives a particular error.
     */
    Sql fails(@Nonnull String expected) {
      Objects.requireNonNull(expected);
      tester.assertExceptionIsThrown(toSql(true), expected);
      return this;
    }

    /**
     * Checks that a SQL expression fails, giving an {@code expected} error,
     * if {@code b} is true, otherwise succeeds.
     */
    Sql failsIf(boolean b, String expected) {
      if (b) {
        fails(expected);
      } else {
        ok();
      }
      return this;
    }

    /**
     * Checks that a query returns a row of the expected type. For example,
     *
     *

     *   sql("select empno, name from emp")

     *     .type("{EMPNO INTEGER NOT NULL, NAME VARCHAR(10) NOT NULL}");
     *

     *
     * @param expectedType Expected row type
     */
    public Sql type(String expectedType) {
      tester.checkResultType(sql, expectedType);
      return this;
    }

    /**
     * Checks that a query returns a single column, and that the column has the
     * expected type. For example,
     *
     *

     * sql("SELECT empno FROM Emp").columnType("INTEGER NOT NULL");
     *

     *
     * @param expectedType Expected type, including nullability
     */
    public Sql columnType(String expectedType) {
      tester.checkColumnType(toSql(false), expectedType);
      return this;
    }

    public Sql monotonic(SqlMonotonicity expectedMonotonicity) {
      tester.checkMonotonic(toSql(false), expectedMonotonicity);
      return this;
    }

    public Sql bindType(final String bindType) {
      tester.check(sql, null, parameterRowType ->
          assertThat(parameterRowType.toString(), is(bindType)),
          result -> { });
      return this;
    }

    /** Removes the carets from the SQL string. Useful if you want to run
     * a test once at a conformance level where it fails, then run it again
     * at a conformance level where it succeeds. */
    public Sql sansCarets() {
      return new Sql(tester, sql.replace("^", ""), true, false);
    }

    public void charset(Charset expectedCharset) {
      tester.checkCharset(sql, expectedCharset);
    }

    public void collation(String expectedCollationName,
        SqlCollation.Coercibility expectedCoercibility) {
      tester.checkCollation(sql, expectedCollationName, expectedCoercibility);
    }

    /**
     * Checks if the interval value conversion to milliseconds is valid. For
     * example,
     *
     *

     *   sql("VALUES (INTERVAL '1' Minute)").intervalConv("60000");
     *

     */
    public void intervalConv(String expected) {
      tester.checkIntervalConv(toSql(false), expected);
    }

    public Sql withCaseSensitive(boolean caseSensitive) {
      return withTester(tester -> tester.withCaseSensitive(caseSensitive));
    }

    public Sql withOperatorTable(SqlOperatorTable operatorTable) {
      return withTester(tester -> tester.withOperatorTable(operatorTable));
    }

    public Sql withUnquotedCasing(Casing casing) {
      return withTester(tester -> tester.withUnquotedCasing(casing));
    }

    private SqlTester addTransform(SqlTester tester, UnaryOperator<SqlValidator> after) {
      return this.tester.withValidatorTransform(transform ->
          validator -> after.apply(transform.apply(validator)));
    }

    public Sql withValidatorIdentifierExpansion(boolean expansion) {
      final UnaryOperator<SqlValidator> after = sqlValidator ->
          sqlValidator.transform(config -> config.withIdentifierExpansion(expansion));
      return withTester(tester -> addTransform(tester, after));
    }

    public Sql withValidatorCallRewrite(boolean rewrite) {
      final UnaryOperator<SqlValidator> after = sqlValidator ->
          sqlValidator.transform(config -> config.withCallRewrite(rewrite));
      return withTester(tester -> addTransform(tester, after));
    }

    public Sql withValidatorColumnReferenceExpansion(boolean expansion) {
      final UnaryOperator<SqlValidator> after = sqlValidator ->
          sqlValidator.transform(config -> config.withColumnReferenceExpansion(expansion));
      return withTester(tester -> addTransform(tester, after));
    }

    public Sql rewritesTo(String expected) {
      tester.checkRewrite(toSql(false), expected);
      return this;
    }
  }

  /**
   * Enables to configure {@link #tester} behavior on a per-test basis.
   * {@code tester} object is created in the test object constructor, and
   * there's no trivial way to override its features.
   *
   *
This JUnit rule enables post-process test object on a per test method
   * basis.
   */
  public static class LexConfiguration implements BeforeEachCallback {
    @Override public void beforeEach(ExtensionContext context) {
      context.getElement()
          .flatMap(element -> AnnotationSupport.findAnnotation(element, WithLex.class))
          .ifPresent(lex -> {
            SqlValidatorTestCase tc = (SqlValidatorTestCase) context.getTestInstance().get();
            SqlTester tester = tc.tester;
            tester = tester.withLex(lex.value());
            tc.tester = tester;
          });
    }
  }
}

{'repo_name': 'apache/calcite', 'stars': '1934', 'repo_language': 'Java', 'file_name': 'ServerParserTest.java', 'mime_type': 'text/x-java', 'hash': -8804614573280782683, 'source_dataset': 'data'}


      SUBROUTINE OFFSET_alpha_2_5_7_1(l_a_offset,k_a_offset,size)
C     $Id$
C     This is a Fortran77 program generated by Tensor Contraction Engine v.1.0
C     Copyright (c) Battelle & Pacific Northwest National Laboratory (2002)
C     i2 ( p13 p4 )_vtra
      IMPLICIT NONE
#include "global.fh"
#include "mafdecls.fh"
#include "sym.fh"
#include "errquit.fh"
#include "tce.fh"
      INTEGER l_a_offset
      INTEGER k_a_offset
      INTEGER size
      INTEGER length
      INTEGER addr
      INTEGER p13b
      INTEGER p4b
      length = 0
      DO p13b = noab+1,noab+nvab
      DO p4b = noab+1,noab+nvab
      IF (int_mb(k_spin+p13b-1) .eq. int_mb(k_spin+p4b-1)) THEN
      IF (ieor(int_mb(k_sym+p13b-1),int_mb(k_sym+p4b-1)) .eq. ieor(irrep
     &_v,irrep_tra)) THEN
      IF ((.not.restricted).or.(int_mb(k_spin+p13b-1)+int_mb(k_spin+p4b-
     &1).ne.4)) THEN
      length = length + 1
      END IF
      END IF
      END IF
      END DO
      END DO
      IF (.not.MA_PUSH_GET(mt_int,2*length+1,'noname',l_a_offset,k_a_off
     &set)) CALL ERRQUIT('alpha_2_5_7_1',0,MA_ERR)
      int_mb(k_a_offset) = length
      addr = 0
      size = 0
      DO p13b = noab+1,noab+nvab
      DO p4b = noab+1,noab+nvab
      IF (int_mb(k_spin+p13b-1) .eq. int_mb(k_spin+p4b-1)) THEN
      IF (ieor(int_mb(k_sym+p13b-1),int_mb(k_sym+p4b-1)) .eq. ieor(irrep
     &_v,irrep_tra)) THEN
      IF ((.not.restricted).or.(int_mb(k_spin+p13b-1)+int_mb(k_spin+p4b-
     &1).ne.4)) THEN
      addr = addr + 1
      int_mb(k_a_offset+addr) = p4b - noab - 1 + nvab * (p13b - noab - 1
     &)
      int_mb(k_a_offset+length+addr) = size
      size = size + int_mb(k_range+p13b-1) * int_mb(k_range+p4b-1)
      END IF
      END IF
      END IF
      END DO
      END DO
      RETURN
      END

{'repo_name': 'nwchemgit/nwchem', 'stars': '180', 'repo_language': 'Fortran', 'file_name': 'doqmtests', 'mime_type': 'text/x-shellscript', 'hash': -6736242984713472671, 'source_dataset': 'data'}


// Copyright 2015 The etcd Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package fileutil

import (
	"os"
	"path"
	"sort"
	"strings"
	"time"
)

func PurgeFile(dirname string, suffix string, max uint, interval time.Duration, stop <-chan struct{}) <-chan error {
	return purgeFile(dirname, suffix, max, interval, stop, nil)
}

// purgeFile is the internal implementation for PurgeFile which can post purged files to purgec if non-nil.
func purgeFile(dirname string, suffix string, max uint, interval time.Duration, stop <-chan struct{}, purgec chan<- string) <-chan error {
	errC := make(chan error, 1)
	go func() {
		for {
			fnames, err := ReadDir(dirname)
			if err != nil {
				errC <- err
				return
			}
			newfnames := make([]string, 0)
			for _, fname := range fnames {
				if strings.HasSuffix(fname, suffix) {
					newfnames = append(newfnames, fname)
				}
			}
			sort.Strings(newfnames)
			fnames = newfnames
			for len(newfnames) > int(max) {
				f := path.Join(dirname, newfnames[0])
				l, err := TryLockFile(f, os.O_WRONLY, PrivateFileMode)
				if err != nil {
					break
				}
				if err = os.Remove(f); err != nil {
					errC <- err
					return
				}
				if err = l.Close(); err != nil {
					plog.Errorf("error unlocking %s when purging file (%v)", l.Name(), err)
					errC <- err
					return
				}
				plog.Infof("purged file %s successfully", f)
				newfnames = newfnames[1:]
			}
			if purgec != nil {
				for i := 0; i < len(fnames)-len(newfnames); i++ {
					purgec <- fnames[i]
				}
			}
			select {
			case <-time.After(interval):
			case <-stop:
				return
			}
		}
	}()
	return errC
}

{'repo_name': 'elastos/Elastos', 'stars': '101', 'repo_language': 'Go', 'file_name': 'codereview.cfg', 'mime_type': 'text/plain', 'hash': -3178518288467890489, 'source_dataset': 'data'}


#!/usr/bin/env bash

set -eux

source hack/lib/test_lib.sh
source hack/lib/image_lib.sh

ROOTDIR="$(pwd)"
TMPDIR="$(mktemp -d)"
trap_add 'rm -rf $TMPDIR' EXIT

# build scorecard test kuttl image
WD="$(dirname "$(pwd)")"
GOOS=linux CGO_ENABLED=0 \
  go build \
  -gcflags "all=-trimpath=${WD}" \
  -asmflags "all=-trimpath=${WD}" \
  -o $TMPDIR/scorecard-test-kuttl \
  images/scorecard-test-kuttl/cmd/test-kuttl/main.go

# Build base image
pushd $TMPDIR
cp -r $ROOTDIR/images/scorecard-test-kuttl/bin .

docker build -f $ROOTDIR/images/scorecard-test-kuttl/Dockerfile -t $1 .
# If using a kind cluster, load the image into all nodes.
setup_envs $tmp_sdk_root
load_image_if_kind "$1"
popd

{'repo_name': 'operator-framework/operator-sdk', 'stars': '3685', 'repo_language': 'Go', 'file_name': 'norms-to-follow.md', 'mime_type': 'text/plain', 'hash': -1547197330507465232, 'source_dataset': 'data'}


<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">

<title>weka.core.converters Class Hierarchy</title>
<meta name="date" content="2018-06-05">
<link rel="stylesheet" type="text/css" href="../../../stylesheet.css" title="Style">

JavaScript is disabled on your browser.

Skip navigation links

Overview

Package

Class

Tree

Deprecated

Index

Help

Prev

Next

Frames

No Frames

All Classes

Hierarchy For Package weka.core.converters

Package Hierarchies:

All Packages

Class Hierarchy

java.lang.Object

weka.core.converters.AbstractLoader (implements weka.core.converters.Loader)

weka.core.converters.AbstractFileLoader (implements weka.core.EnvironmentHandler, weka.core.converters.FileSourcedConverter)

weka.core.converters.Word2VecLoader (implements weka.core.converters.BatchConverter)

Skip navigation links

Overview

Package

Class

Tree

Deprecated

Index

Help

Prev

Next

Frames

No Frames

All Classes

{'repo_name': 'Waikato/wekaDeeplearning4j', 'stars': '108', 'repo_language': 'Java', 'file_name': 'package-summary.html', 'mime_type': 'text/html', 'hash': -3018570489712519762, 'source_dataset': 'data'}


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta name="author" content="Emanuel Eichhammer" />
<meta name="copyright" content="(C) 2011-2016 Emanuel Eichhammer" />
<title>Data Fields - Functions</title>
<link href="qcp.css" rel="stylesheet" type="text/css"/>

Main Page ·
Class Overview ·
Hierarchy ·
All Classes ·
Special Pages

- w -

wheelEvent()
: QCPAxis
, QCPAxisRect
, QCPColorScale
, QCPLayerable
, QCustomPlot

width()
: QCPAxisRect

{'repo_name': 'CieNTi/serial_port_plotter', 'stars': '104', 'repo_language': 'C++', 'file_name': 'README.md', 'mime_type': 'text/plain', 'hash': -7331790527521989652, 'source_dataset': 'data'}


//===-- SIInsertSkips.cpp - Use predicates for control flow ---------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
//
/// \file
/// This pass inserts branches on the 0 exec mask over divergent branches
/// branches when it's expected that jumping over the untaken control flow will
/// be cheaper than having every workitem no-op through it.
//
//===----------------------------------------------------------------------===//

#include "AMDGPU.h"
#include "AMDGPUSubtarget.h"
#include "SIInstrInfo.h"
#include "SIMachineFunctionInfo.h"
#include "MCTargetDesc/AMDGPUMCTargetDesc.h"
#include "llvm/ADT/SmallVector.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/CodeGen/MachineBasicBlock.h"
#include "llvm/CodeGen/MachineFunction.h"
#include "llvm/CodeGen/MachineFunctionPass.h"
#include "llvm/CodeGen/MachineInstr.h"
#include "llvm/CodeGen/MachineInstrBuilder.h"
#include "llvm/CodeGen/MachineOperand.h"
#include "llvm/IR/CallingConv.h"
#include "llvm/IR/DebugLoc.h"
#include "llvm/MC/MCAsmInfo.h"
#include "llvm/Pass.h"
#include "llvm/Support/CommandLine.h"
#include "llvm/Target/TargetMachine.h"
#include <cassert>
#include <cstdint>
#include <iterator>

using namespace llvm;

#define DEBUG_TYPE "si-insert-skips"

static cl::opt<unsigned> SkipThresholdFlag(
  "amdgpu-skip-threshold",
  cl::desc("Number of instructions before jumping over divergent control flow"),
  cl::init(12), cl::Hidden);

namespace {

class SIInsertSkips : public MachineFunctionPass {
private:
  const SIRegisterInfo *TRI = nullptr;
  const SIInstrInfo *TII = nullptr;
  unsigned SkipThreshold = 0;

  bool shouldSkip(const MachineBasicBlock &From,
                  const MachineBasicBlock &To) const;

  bool skipIfDead(MachineInstr &MI, MachineBasicBlock &NextBB);

  void kill(MachineInstr &MI);

  MachineBasicBlock *insertSkipBlock(MachineBasicBlock &MBB,
                                     MachineBasicBlock::iterator I) const;

  bool skipMaskBranch(MachineInstr &MI, MachineBasicBlock &MBB);

  bool optimizeVccBranch(MachineInstr &MI) const;

public:
  static char ID;

  SIInsertSkips() : MachineFunctionPass(ID) {}

  bool runOnMachineFunction(MachineFunction &MF) override;

  StringRef getPassName() const override {
    return "SI insert s_cbranch_execz instructions";
  }

  void getAnalysisUsage(AnalysisUsage &AU) const override {
    MachineFunctionPass::getAnalysisUsage(AU);
  }
};

} // end anonymous namespace

char SIInsertSkips::ID = 0;

INITIALIZE_PASS(SIInsertSkips, DEBUG_TYPE,
                "SI insert s_cbranch_execz instructions", false, false)

char &llvm::SIInsertSkipsPassID = SIInsertSkips::ID;

static bool opcodeEmitsNoInsts(const MachineInstr &MI) {
  if (MI.isMetaInstruction())
    return true;

  // Handle target specific opcodes.
  switch (MI.getOpcode()) {
  case AMDGPU::SI_MASK_BRANCH:
    return true;
  default:
    return false;
  }
}

bool SIInsertSkips::shouldSkip(const MachineBasicBlock &From,
                               const MachineBasicBlock &To) const {
  unsigned NumInstr = 0;
  const MachineFunction *MF = From.getParent();

  for (MachineFunction::const_iterator MBBI(&From), ToI(&To), End = MF->end();
       MBBI != End && MBBI != ToI; ++MBBI) {
    const MachineBasicBlock &MBB = *MBBI;

    for (MachineBasicBlock::const_iterator I = MBB.begin(), E = MBB.end();
         NumInstr < SkipThreshold && I != E; ++I) {
      if (opcodeEmitsNoInsts(*I))
        continue;

      // FIXME: Since this is required for correctness, this should be inserted
      // during SILowerControlFlow.

      // When a uniform loop is inside non-uniform control flow, the branch
      // leaving the loop might be an S_CBRANCH_VCCNZ, which is never taken
      // when EXEC = 0. We should skip the loop lest it becomes infinite.
      if (I->getOpcode() == AMDGPU::S_CBRANCH_VCCNZ ||
          I->getOpcode() == AMDGPU::S_CBRANCH_VCCZ)
        return true;

      if (TII->hasUnwantedEffectsWhenEXECEmpty(*I))
        return true;

      // These instructions are potentially expensive even if EXEC = 0.
      if (TII->isSMRD(*I) || TII->isVMEM(*I) || TII->isFLAT(*I) ||
          I->getOpcode() == AMDGPU::S_WAITCNT)
        return true;

      ++NumInstr;
      if (NumInstr >= SkipThreshold)
        return true;
    }
  }

  return false;
}

bool SIInsertSkips::skipIfDead(MachineInstr &MI, MachineBasicBlock &NextBB) {
  MachineBasicBlock &MBB = *MI.getParent();
  MachineFunction *MF = MBB.getParent();

  if (MF->getFunction().getCallingConv() != CallingConv::AMDGPU_PS ||
      !shouldSkip(MBB, MBB.getParent()->back()))
    return false;

  MachineBasicBlock *SkipBB = insertSkipBlock(MBB, MI.getIterator());

  const DebugLoc &DL = MI.getDebugLoc();

  // If the exec mask is non-zero, skip the next two instructions
  BuildMI(&MBB, DL, TII->get(AMDGPU::S_CBRANCH_EXECNZ))
    .addMBB(&NextBB);

  MachineBasicBlock::iterator Insert = SkipBB->begin();

  // Exec mask is zero: Export to NULL target...
  BuildMI(*SkipBB, Insert, DL, TII->get(AMDGPU::EXP_DONE))
    .addImm(0x09) // V_008DFC_SQ_EXP_NULL
    .addReg(AMDGPU::VGPR0, RegState::Undef)
    .addReg(AMDGPU::VGPR0, RegState::Undef)
    .addReg(AMDGPU::VGPR0, RegState::Undef)
    .addReg(AMDGPU::VGPR0, RegState::Undef)
    .addImm(1)  // vm
    .addImm(0)  // compr
    .addImm(0); // en

  // ... and terminate wavefront.
  BuildMI(*SkipBB, Insert, DL, TII->get(AMDGPU::S_ENDPGM)).addImm(0);

  return true;
}

void SIInsertSkips::kill(MachineInstr &MI) {
  MachineBasicBlock &MBB = *MI.getParent();
  DebugLoc DL = MI.getDebugLoc();

  switch (MI.getOpcode()) {
  case AMDGPU::SI_KILL_F32_COND_IMM_TERMINATOR: {
    unsigned Opcode = 0;

    // The opcodes are inverted because the inline immediate has to be
    // the first operand, e.g. from "x < imm" to "imm > x"
    switch (MI.getOperand(2).getImm()) {
    case ISD::SETOEQ:
    case ISD::SETEQ:
      Opcode = AMDGPU::V_CMPX_EQ_F32_e64;
      break;
    case ISD::SETOGT:
    case ISD::SETGT:
      Opcode = AMDGPU::V_CMPX_LT_F32_e64;
      break;
    case ISD::SETOGE:
    case ISD::SETGE:
      Opcode = AMDGPU::V_CMPX_LE_F32_e64;
      break;
    case ISD::SETOLT:
    case ISD::SETLT:
      Opcode = AMDGPU::V_CMPX_GT_F32_e64;
      break;
    case ISD::SETOLE:
    case ISD::SETLE:
      Opcode = AMDGPU::V_CMPX_GE_F32_e64;
      break;
    case ISD::SETONE:
    case ISD::SETNE:
      Opcode = AMDGPU::V_CMPX_LG_F32_e64;
      break;
    case ISD::SETO:
      Opcode = AMDGPU::V_CMPX_O_F32_e64;
      break;
    case ISD::SETUO:
      Opcode = AMDGPU::V_CMPX_U_F32_e64;
      break;
    case ISD::SETUEQ:
      Opcode = AMDGPU::V_CMPX_NLG_F32_e64;
      break;
    case ISD::SETUGT:
      Opcode = AMDGPU::V_CMPX_NGE_F32_e64;
      break;
    case ISD::SETUGE:
      Opcode = AMDGPU::V_CMPX_NGT_F32_e64;
      break;
    case ISD::SETULT:
      Opcode = AMDGPU::V_CMPX_NLE_F32_e64;
      break;
    case ISD::SETULE:
      Opcode = AMDGPU::V_CMPX_NLT_F32_e64;
      break;
    case ISD::SETUNE:
      Opcode = AMDGPU::V_CMPX_NEQ_F32_e64;
      break;
    default:
      llvm_unreachable("invalid ISD:SET cond code");
    }

    const GCNSubtarget &ST = MBB.getParent()->getSubtarget<GCNSubtarget>();
    if (ST.hasNoSdstCMPX())
      Opcode = AMDGPU::getVCMPXNoSDstOp(Opcode);

    assert(MI.getOperand(0).isReg());

    if (TRI->isVGPR(MBB.getParent()->getRegInfo(),
                    MI.getOperand(0).getReg())) {
      Opcode = AMDGPU::getVOPe32(Opcode);
      BuildMI(MBB, &MI, DL, TII->get(Opcode))
          .add(MI.getOperand(1))
          .add(MI.getOperand(0));
    } else {
      auto I = BuildMI(MBB, &MI, DL, TII->get(Opcode));
      if (!ST.hasNoSdstCMPX())
        I.addReg(AMDGPU::VCC, RegState::Define);

      I.addImm(0)  // src0 modifiers
        .add(MI.getOperand(1))
        .addImm(0)  // src1 modifiers
        .add(MI.getOperand(0));

      I.addImm(0);  // omod
    }
    break;
  }
  case AMDGPU::SI_KILL_I1_TERMINATOR: {
    const MachineFunction *MF = MI.getParent()->getParent();
    const GCNSubtarget &ST = MF->getSubtarget<GCNSubtarget>();
    unsigned Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
    const MachineOperand &Op = MI.getOperand(0);
    int64_t KillVal = MI.getOperand(1).getImm();
    assert(KillVal == 0 || KillVal == -1);

    // Kill all threads if Op0 is an immediate and equal to the Kill value.
    if (Op.isImm()) {
      int64_t Imm = Op.getImm();
      assert(Imm == 0 || Imm == -1);

      if (Imm == KillVal)
        BuildMI(MBB, &MI, DL, TII->get(ST.isWave32() ? AMDGPU::S_MOV_B32
                                                     : AMDGPU::S_MOV_B64), Exec)
          .addImm(0);
      break;
    }

    unsigned Opcode = KillVal ? AMDGPU::S_ANDN2_B64 : AMDGPU::S_AND_B64;
    if (ST.isWave32())
      Opcode = KillVal ? AMDGPU::S_ANDN2_B32 : AMDGPU::S_AND_B32;
    BuildMI(MBB, &MI, DL, TII->get(Opcode), Exec)
        .addReg(Exec)
        .add(Op);
    break;
  }
  default:
    llvm_unreachable("invalid opcode, expected SI_KILL_*_TERMINATOR");
  }
}

MachineBasicBlock *SIInsertSkips::insertSkipBlock(
  MachineBasicBlock &MBB, MachineBasicBlock::iterator I) const {
  MachineFunction *MF = MBB.getParent();

  MachineBasicBlock *SkipBB = MF->CreateMachineBasicBlock();
  MachineFunction::iterator MBBI(MBB);
  ++MBBI;

  MF->insert(MBBI, SkipBB);
  MBB.addSuccessor(SkipBB);

  return SkipBB;
}

// Returns true if a branch over the block was inserted.
bool SIInsertSkips::skipMaskBranch(MachineInstr &MI,
                                   MachineBasicBlock &SrcMBB) {
  MachineBasicBlock *DestBB = MI.getOperand(0).getMBB();

  if (!shouldSkip(**SrcMBB.succ_begin(), *DestBB))
    return false;

  const DebugLoc &DL = MI.getDebugLoc();
  MachineBasicBlock::iterator InsPt = std::next(MI.getIterator());

  BuildMI(SrcMBB, InsPt, DL, TII->get(AMDGPU::S_CBRANCH_EXECZ))
    .addMBB(DestBB);

  return true;
}

bool SIInsertSkips::optimizeVccBranch(MachineInstr &MI) const {
  // Match:
  // sreg = -1
  // vcc = S_AND_B64 exec, sreg
  // S_CBRANCH_VCC[N]Z
  // =>
  // S_CBRANCH_EXEC[N]Z
  bool Changed = false;
  MachineBasicBlock &MBB = *MI.getParent();
  const GCNSubtarget &ST = MBB.getParent()->getSubtarget<GCNSubtarget>();
  const bool IsWave32 = ST.isWave32();
  const unsigned CondReg = TRI->getVCC();
  const unsigned ExecReg = IsWave32 ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
  const unsigned And = IsWave32 ? AMDGPU::S_AND_B32 : AMDGPU::S_AND_B64;

  MachineBasicBlock::reverse_iterator A = MI.getReverseIterator(),
                                      E = MBB.rend();
  bool ReadsCond = false;
  unsigned Threshold = 5;
  for (++A ; A != E ; ++A) {
    if (!--Threshold)
      return false;
    if (A->modifiesRegister(ExecReg, TRI))
      return false;
    if (A->modifiesRegister(CondReg, TRI)) {
      if (!A->definesRegister(CondReg, TRI) || A->getOpcode() != And)
        return false;
      break;
    }
    ReadsCond |= A->readsRegister(CondReg, TRI);
  }
  if (A == E)
    return false;

  MachineOperand &Op1 = A->getOperand(1);
  MachineOperand &Op2 = A->getOperand(2);
  if (Op1.getReg() != ExecReg && Op2.isReg() && Op2.getReg() == ExecReg) {
    TII->commuteInstruction(*A);
    Changed = true;
  }
  if (Op1.getReg() != ExecReg)
    return Changed;
  if (Op2.isImm() && Op2.getImm() != -1)
    return Changed;

  unsigned SReg = AMDGPU::NoRegister;
  if (Op2.isReg()) {
    SReg = Op2.getReg();
    auto M = std::next(A);
    bool ReadsSreg = false;
    for ( ; M != E ; ++M) {
      if (M->definesRegister(SReg, TRI))
        break;
      if (M->modifiesRegister(SReg, TRI))
        return Changed;
      ReadsSreg |= M->readsRegister(SReg, TRI);
    }
    if (M == E ||
        !M->isMoveImmediate() ||
        !M->getOperand(1).isImm() ||
        M->getOperand(1).getImm() != -1)
      return Changed;
    // First if sreg is only used in and instruction fold the immediate
    // into that and.
    if (!ReadsSreg && Op2.isKill()) {
      A->getOperand(2).ChangeToImmediate(-1);
      M->eraseFromParent();
    }
  }

  if (!ReadsCond && A->registerDefIsDead(AMDGPU::SCC) &&
      MI.killsRegister(CondReg, TRI))
    A->eraseFromParent();

  bool IsVCCZ = MI.getOpcode() == AMDGPU::S_CBRANCH_VCCZ;
  if (SReg == ExecReg) {
    if (IsVCCZ) {
      MI.eraseFromParent();
      return true;
    }
    MI.setDesc(TII->get(AMDGPU::S_BRANCH));
  } else {
    MI.setDesc(TII->get(IsVCCZ ? AMDGPU::S_CBRANCH_EXECZ
                               : AMDGPU::S_CBRANCH_EXECNZ));
  }

  MI.RemoveOperand(MI.findRegisterUseOperandIdx(CondReg, false /*Kill*/, TRI));
  MI.addImplicitDefUseOperands(*MBB.getParent());

  return true;
}

bool SIInsertSkips::runOnMachineFunction(MachineFunction &MF) {
  const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
  TII = ST.getInstrInfo();
  TRI = &TII->getRegisterInfo();
  SkipThreshold = SkipThresholdFlag;

  bool HaveKill = false;
  bool MadeChange = false;

  // Track depth of exec mask, divergent branches.
  SmallVector<MachineBasicBlock *, 16> ExecBranchStack;

  MachineFunction::iterator NextBB;

  MachineBasicBlock *EmptyMBBAtEnd = nullptr;

  for (MachineFunction::iterator BI = MF.begin(), BE = MF.end();
       BI != BE; BI = NextBB) {
    NextBB = std::next(BI);
    MachineBasicBlock &MBB = *BI;
    bool HaveSkipBlock = false;

    if (!ExecBranchStack.empty() && ExecBranchStack.back() == &MBB) {
      // Reached convergence point for last divergent branch.
      ExecBranchStack.pop_back();
    }

    if (HaveKill && ExecBranchStack.empty()) {
      HaveKill = false;

      // TODO: Insert skip if exec is 0?
    }

    MachineBasicBlock::iterator I, Next;
    for (I = MBB.begin(); I != MBB.end(); I = Next) {
      Next = std::next(I);

      MachineInstr &MI = *I;

      switch (MI.getOpcode()) {
      case AMDGPU::SI_MASK_BRANCH:
        ExecBranchStack.push_back(MI.getOperand(0).getMBB());
        MadeChange |= skipMaskBranch(MI, MBB);
        break;

      case AMDGPU::S_BRANCH:
        // Optimize out branches to the next block.
        // FIXME: Shouldn't this be handled by BranchFolding?
        if (MBB.isLayoutSuccessor(MI.getOperand(0).getMBB())) {
          MI.eraseFromParent();
        } else if (HaveSkipBlock) {
          // Remove the given unconditional branch when a skip block has been
          // inserted after the current one and let skip the two instructions
          // performing the kill if the exec mask is non-zero.
          MI.eraseFromParent();
        }
        break;

      case AMDGPU::SI_KILL_F32_COND_IMM_TERMINATOR:
      case AMDGPU::SI_KILL_I1_TERMINATOR:
        MadeChange = true;
        kill(MI);

        if (ExecBranchStack.empty()) {
          if (NextBB != BE && skipIfDead(MI, *NextBB)) {
            HaveSkipBlock = true;
            NextBB = std::next(BI);
            BE = MF.end();
          }
        } else {
          HaveKill = true;
        }

        MI.eraseFromParent();
        break;

      case AMDGPU::SI_RETURN_TO_EPILOG:
        // FIXME: Should move somewhere else
        assert(!MF.getInfo<SIMachineFunctionInfo>()->returnsVoid());

        // Graphics shaders returning non-void shouldn't contain S_ENDPGM,
        // because external bytecode will be appended at the end.
        if (BI != --MF.end() || I != MBB.getFirstTerminator()) {
          // SI_RETURN_TO_EPILOG is not the last instruction. Add an empty block at
          // the end and jump there.
          if (!EmptyMBBAtEnd) {
            EmptyMBBAtEnd = MF.CreateMachineBasicBlock();
            MF.insert(MF.end(), EmptyMBBAtEnd);
          }

          MBB.addSuccessor(EmptyMBBAtEnd);
          BuildMI(*BI, I, MI.getDebugLoc(), TII->get(AMDGPU::S_BRANCH))
            .addMBB(EmptyMBBAtEnd);
          I->eraseFromParent();
        }
        break;

      case AMDGPU::S_CBRANCH_VCCZ:
      case AMDGPU::S_CBRANCH_VCCNZ:
        MadeChange |= optimizeVccBranch(MI);
        break;

      default:
        break;
      }
    }
  }

  return MadeChange;
}

{'repo_name': 'earl/llvm-mirror', 'stars': '122', 'repo_language': 'LLVM', 'file_name': 'registers_readonly.s', 'mime_type': 'text/plain', 'hash': 1569518574562819064, 'source_dataset': 'data'}


"""An implementation of DSSM, Deep Structured Semantic Model."""
import typing

import torch
import torch.nn.functional as F

from matchzoo import preprocessors
from matchzoo.engine.param_table import ParamTable
from matchzoo.engine.param import Param
from matchzoo.engine.base_model import BaseModel
from matchzoo.engine.base_preprocessor import BasePreprocessor

class DSSM(BaseModel):
    """
    Deep structured semantic model.

    Examples:
        >>> model = DSSM()
        >>> model.params['mlp_num_layers'] = 3
        >>> model.params['mlp_num_units'] = 300
        >>> model.params['mlp_num_fan_out'] = 128
        >>> model.params['mlp_activation_func'] = 'relu'
        >>> model.guess_and_fill_missing_params(verbose=0)
        >>> model.build()

    """

    @classmethod
    def get_default_params(cls) -> ParamTable:
        """:return: model default parameters."""
        params = super().get_default_params(with_multi_layer_perceptron=True)
        params.add(Param(name='vocab_size', value=419,
                         desc="Size of vocabulary."))
        return params

    @classmethod
    def get_default_preprocessor(
        cls,
        truncated_mode: str = 'pre',
        truncated_length_left: typing.Optional[int] = None,
        truncated_length_right: typing.Optional[int] = None,
        filter_mode: str = 'df',
        filter_low_freq: float = 1,
        filter_high_freq: float = float('inf'),
        remove_stop_words: bool = False,
        ngram_size: typing.Optional[int] = 3,
    ) -> BasePreprocessor:
        """
        Model default preprocessor.

        The preprocessor's transform should produce a correctly shaped data
        pack that can be used for training.

        :return: Default preprocessor.
        """
        return preprocessors.BasicPreprocessor(
            truncated_mode=truncated_mode,
            truncated_length_left=truncated_length_left,
            truncated_length_right=truncated_length_right,
            filter_mode=filter_mode,
            filter_low_freq=filter_low_freq,
            filter_high_freq=filter_high_freq,
            remove_stop_words=remove_stop_words,
            ngram_size=ngram_size
        )

    @classmethod
    def get_default_padding_callback(cls):
        """:return: Default padding callback."""
        return None

    def build(self):
        """
        Build model structure.

        DSSM use Siamese arthitecture.
        """
        self.mlp_left = self._make_multi_layer_perceptron_layer(
            self._params['vocab_size']
        )
        self.mlp_right = self._make_multi_layer_perceptron_layer(
            self._params['vocab_size']
        )
        self.out = self._make_output_layer(1)

    def forward(self, inputs):
        """Forward."""
        # Process left & right input.
        input_left, input_right = inputs['ngram_left'], inputs['ngram_right']
        input_left = self.mlp_left(input_left)
        input_right = self.mlp_right(input_right)

        # Dot product with cosine similarity.
        x = F.cosine_similarity(input_left, input_right)

        out = self.out(x.unsqueeze(dim=1))
        return out

{'repo_name': 'NTMC-Community/MatchZoo-py', 'stars': '254', 'repo_language': 'Python', 'file_name': 'test_models.py', 'mime_type': 'text/x-python', 'hash': 3625069246284843668, 'source_dataset': 'data'}


framework module CheckMarkView {
  umbrella header "CheckMarkView-umbrella.h"

  export *
  module * { export * }
}

{'repo_name': 'shu223/iOS-Depth-Sampler', 'stars': '815', 'repo_language': 'Swift', 'file_name': 'IDEWorkspaceChecks.plist', 'mime_type': 'text/xml', 'hash': -7376078498490531684, 'source_dataset': 'data'}


// Copyright 2015 The etcd Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package command

import (
	"errors"
	"os"
	"time"

	"github.com/coreos/etcd/client"
	"github.com/urfave/cli"
)

// NewUpdateCommand returns the CLI command for "update".
func NewUpdateCommand() cli.Command {
	return cli.Command{
		Name:      "update",
		Usage:     "update an existing key with a given value",
		ArgsUsage: "<key> <value>",
		Flags: []cli.Flag{
			cli.IntFlag{Name: "ttl", Value: 0, Usage: "key time-to-live"},
		},
		Action: func(c *cli.Context) error {
			updateCommandFunc(c, mustNewKeyAPI(c))
			return nil
		},
	}
}

// updateCommandFunc executes the "update" command.
func updateCommandFunc(c *cli.Context, ki client.KeysAPI) {
	if len(c.Args()) == 0 {
		handleError(ExitBadArgs, errors.New("key required"))
	}
	key := c.Args()[0]
	value, err := argOrStdin(c.Args(), os.Stdin, 1)
	if err != nil {
		handleError(ExitBadArgs, errors.New("value required"))
	}

	ttl := c.Int("ttl")

	ctx, cancel := contextWithTotalTimeout(c)
	resp, err := ki.Set(ctx, key, value, &client.SetOptions{TTL: time.Duration(ttl) * time.Second, PrevExist: client.PrevExist})
	cancel()
	if err != nil {
		handleError(ExitServerError, err)
	}

	printResponseKey(resp, c.GlobalString("output"))
}

{'repo_name': 'ehazlett/interlock', 'stars': '961', 'repo_language': 'Go', 'file_name': 'utils_test.go', 'mime_type': 'text/plain', 'hash': -6636244456924739472, 'source_dataset': 'data'}


import createIcon, { JSX } from './utils/createIcon'

export const jsx: JSX = {
  type: 'svg',
  props: {
    xmlns: 'http://www.w3.org/2000/svg',
    viewBox: '0 0 24 24'
  },
  children: [
    {
      type: 'g',
      props: null,
      children: [
        {
          type: 'path',
          props: {
            fill: 'none',
            d: 'M0 0h24v24H0z'
          },
          children: []
        },
        {
          type: 'path',
          props: {
            d:
              'M23 12l-7.071 7.071-1.414-1.414L20.172 12l-5.657-5.657 1.414-1.414L23 12zM3.828 12l5.657 5.657-1.414 1.414L1 12l7.071-7.071 1.414 1.414L3.828 12z'
          },
          children: []
        }
      ]
    }
  ]
}

export default createIcon(jsx, 'CodeFill')

{'repo_name': 'fluent-org/fluent-windows', 'stars': '110', 'repo_language': 'TypeScript', 'file_name': 'config.yml', 'mime_type': 'text/plain', 'hash': 3916779581352860525, 'source_dataset': 'data'}


require 'mastodon/version'

module Mastodon
  class Client
    DEFAULT_TIMEOUT = {
      connect: 2,
      read: 5,
      write: 20,
    }.freeze

    attr_reader :base_url, :bearer_token, :timeout

    # @param options [Hash]
    # @option options :base_url [String] URL of the instance you want to connect to
    # @option options :bearer_token [String] OAuth access token for your authenticated user
    def initialize(options = {})
      @base_url     = options[:base_url]
      @bearer_token = options[:bearer_token]
      @timeout      = DEFAULT_TIMEOUT.merge(options[:timeout] || {})
    end

    # User agent of the client
    # @return [String]
    def user_agent
      @user_agent ||= "MastodonRubyGem/#{Mastodon::Version}"
    end
  end
end

{'repo_name': 'tootsuite/mastodon-api', 'stars': '117', 'repo_language': 'Ruby', 'file_name': 'filters_change.rb', 'mime_type': 'text/x-ruby', 'hash': -4864772570249741538, 'source_dataset': 'data'}


	<nml_record name="periodic_planar" mode="init" configuration="periodic_planar">
		<nml_option name="config_periodic_planar_vert_levels" type="integer" default_value="100" units="unitless"
			description="Number of vertical levels in periodic_planar. Typical value is 1."
			possible_values="Any positive integer number greater than 0."
		/>
		<nml_option name="config_periodic_planar_bottom_depth" type="real" default_value="2500.0" units="unitless"
			description="Bottom depth."
			possible_values="Any real number"
		/>
		<nml_option name="config_periodic_planar_velocity_strength" type="real" default_value="1.0" units="m^2 s^{-1}"
			description="Strenght of velocity field."
			possible_values="Any real number"
		/>
	</nml_record>
	<var_struct name="scratch" time_levs="0">
		<var name="psiVertex" type="real" dimensions="nVertices" persistence="scratch"
			 default_value="0" units="unitless"
			 description="Temporary space to hold vertex values for psi."
		/>
	</var_struct>

{'repo_name': 'MPAS-Dev/MPAS-Model', 'stars': '129', 'repo_language': 'Fortran', 'file_name': 'testsuite.standard.xml', 'mime_type': 'text/xml', 'hash': 2543724321515646365, 'source_dataset': 'data'}


/*
Copyright (c) 2003-2017, CKSource - Frederico Knabben. All rights reserved.
For licensing, see LICENSE.md or http://ckeditor.com/license
*/
CKEDITOR.plugins.setLang( 'removeformat', 'hu', {
	toolbar: 'Formázás eltávolítása'
} );

{'repo_name': 'LimeSurvey/LimeSurvey', 'stars': '1547', 'repo_language': 'JavaScript', 'file_name': 'plugin.js', 'mime_type': 'text/plain', 'hash': -2931764244994938583, 'source_dataset': 'data'}


<?php

namespace Drupal\Core\Render\Element;

/**
 * Provides a render element where the user supplies an in-line Twig template.
 *
 * Properties:
 * - #template: The inline Twig template used to render the element.
 * - #context: (array) The variables to substitute into the Twig template.
 *   Each variable may be a string or a render array.
 *
 * Usage example:
 * @code
 * $build['hello']  = [
 *   '#type' => 'inline_template',
 *   '#template' => "{% trans %} Hello {% endtrans %} <strong>{{name}}</strong>",
 *   '#context' => [
 *     'name' => $name,
 *   ]
 * ];
 * @endcode
 *
 * @RenderElement("inline_template")
 */
class InlineTemplate extends RenderElement {

  /**
   * {@inheritdoc}
   */
  public function getInfo() {
    $class = get_class($this);
    return [
      '#pre_render' => [
        [$class, 'preRenderInlineTemplate'],
      ],
      '#template' => '',
      '#context' => [],
    ];
  }

  /**
   * Renders a twig string directly.
   *
   * @param array $element
   *
   * @return array
   */
  public static function preRenderInlineTemplate($element) {
    /** @var \Drupal\Core\Template\TwigEnvironment $environment */
    $environment = \Drupal::service('twig');
    $markup = $environment->renderInline($element['#template'], $element['#context']);
    $element['#markup'] = $markup;
    return $element;
  }

}

{'repo_name': 'fossasia/phimpme-drupal', 'stars': '1111', 'repo_language': 'PHP', 'file_name': 'jquery.ui.datepicker.css', 'mime_type': 'text/plain', 'hash': 3189372134819754803, 'source_dataset': 'data'}


package com.google.android.gms.internal;

final class zzfkp extends zzfkn<zzfko, zzfko> {
    zzfkp() {
    }

    final /* synthetic */ void zzb(Object obj, zzfli zzfli) {
        ((zzfko) obj).zza(zzfli);
    }

    final /* synthetic */ Object zzcu(Object obj) {
        return ((zzfhu) obj).zzpph;
    }

    final /* synthetic */ int zzcv(Object obj) {
        return ((zzfko) obj).zzdcc();
    }
}

{'repo_name': 'Hengle/DeepNewdsForAndroid', 'stars': '234', 'repo_language': 'Java', 'file_name': 'androidx.versionedparcelable_versionedparcelable.version', 'mime_type': 'text/plain', 'hash': -6352615410923226137, 'source_dataset': 'data'}


// Copyright (c) 2006-2013, Andrey N. Sabelnikov, www.sabelnikov.net
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are met:
// * Redistributions of source code must retain the above copyright
// notice, this list of conditions and the following disclaimer.
// * Redistributions in binary form must reproduce the above copyright
// notice, this list of conditions and the following disclaimer in the
// documentation and/or other materials provided with the distribution.
// * Neither the name of the Andrey N. Sabelnikov nor the
// names of its contributors may be used to endorse or promote products
// derived from this software without specific prior written permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
// WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
// DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER  BE LIABLE FOR ANY
// DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
// (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
// LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
// ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
// SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//

#ifndef _HTTP_SERVER_CP_H_
#define _HTTP_SERVER_CP_H_

#include "abstract_tcp_server_cp.h"
#include "levin_protocol_handler.h"
namespace epee
{
namespace net_utils
{
	typedef cp_server_impl<levin::protocol_handler> cp_levin_server;
}
}

#endif

{'repo_name': 'monero-project/monero', 'stars': '4414', 'repo_language': 'C++', 'file_name': 'monero_cat.ts', 'mime_type': 'text/xml', 'hash': 2788094879957071155, 'source_dataset': 'data'}


fileFormatVersion: 2
guid: 451dfd61c4a844206866a9e2c3bd8962
MonoImporter:
  externalObjects: {}
  serializedVersion: 2
  defaultReferences: []
  executionOrder: 0
  icon: {instanceID: 0}
  userData:
  assetBundleName:
  assetBundleVariant:

{'repo_name': 'AdamRamberg/unity-atoms', 'stars': '335', 'repo_language': 'C#', 'file_name': 'AudioManager.asset', 'mime_type': 'text/plain', 'hash': -848148265348025683, 'source_dataset': 'data'}