This includes a patch that fixes the weird white borders on tables


d


ddd


I think this broke the Deploy to Heroku button:

![](http://cl.ly/bpJM/Image%202015-06-30%20at%208.00.28%20PM.png)

Best guess is that the trailing comma on line 24 should be removed. JSON is picky like that.


dd


I think I should prefer that spectralline remain unchanged
and that LineList handle 'unknown' all by itself.


I thought inbound rules were always ingress?


See my line notes (I assume you can).


One thing we should check for here is the difference between not specifying the `inbound_rules` attribute and specifying it as `inbound_rules []`.  This has come up recently on the Security Group resource.

In the first case (not specified) we don't change existing inbound rules at all.  In the second case (empty) we delete all inbound rules.


> That's fair, but without trying to sound sassy, I originally saw the using namespace std; from your example code, and from there decided if that's being used, and these sub-modules were going along with it, it'd be okay.

That's definitely true.  There's a discussion question on this; when do you think the `using namespace std` in file scope vs in function scope vs explicit prefixes on each variable should be used? (I guess now that we have a Slack, that's a good place for that discussion. I'll chime in there. :smiley:)

>Also, I was having trouble finding exactly what happens when you have conflicting notation referring to different modules (so include guards don't seem to be relevant here).

These are good questions to ask.

With this, do you mean one namespace defines `_abc` and another namespace defines `_abc`, and you use `using namespace blah;` with each?  In that case, yeah, using `""_abc` would be ambiguous, and you'd get a compiler error.  To explicitly select, you can say `using foo::operator""_abc;` to explicitly say which one you want.

Or do you mean when you have two different header files that declare the same function/type?  As long as the declarations are the same, that's perfectly fine.  If they are different, then we get a compile error.

Using the word _modules_ here probably isn't what you mean.  _Modules_ is something we'll be getting in C++ with C++17, and they provide some really nice benefits.  Now, we use _header files_, which are just files that the preprocessor copies and pastes into your source file—they can contain any sort of code at all!  We also have _namespaces_, which are just groupings of names and don't give build isolation.  Unlike in a lot of other languages, module functionality and namespace functionality is separate; if we had a C++17 module `std.vector` or current C++ header file `<vector>`, it can declare things in any namespace, or even the global namespace.  We'll see why this is an important separation when we start writing some more advanced code in a few weeks.

>And then re your re: about specifying, sorry, could I get you to expand on why that code is not what I'm trying to say there?

[Here's a bit of code](https://gist.github.com/pniedzielski/28adac9b92b3321666f2) that explains what I mean.

I'm going back and forth about the `std`.  It's not too important for a small program like, this definitiely.  Let's play around with this and keep it in mind as we get to larger programs and see what some of the benefits and drawbacks are, okay?  Cheers.


There are many developers for phpMyAdmin, especially students for Google Summer of Code. We try to review code, but strlen() is such a common function that I bet someone will use it, and that at some point we will have to do a massive replacement again.


Another issue that has come up recently is the ability to remove references.  IE, how do I unlink a network_acl from a subnet?  I'm going to propose that we introduce the idea of adding `network_acl false` as the signifier for unlinking the two.  How does this sound?

Paging @jkeiser @randomcamel @metadave


Howdy Archomeda: You're newest update looks real good and works great, however I am getting the Debug Output screen again.
![debug output](https://cloud.githubusercontent.com/assets/12837457/8444841/d9edbf3c-1f63-11e5-9206-f8637b5782a3.png)


Sorry should have linked #423 (not #424)


你在每一台计算机上只需要做一次这些事；它们将会在升级时保留下来。


I really wanted to use `refute_match`, e.g.:

    refute_match(/\f/, stripped)

...but it would always fail.


It seems like you basically repeat
```c++
out << "[";
auto separator = "";
for (const auto & element : object) {
  out << separator;
  out << element;
  separator = ", ";
}
out << "]";
return out;
```
For every container. Why not make this into a macro named something like `PRETTYPRINT_GENERINC_OBJECT_OUTPUT`?


参考第一版:配置工作只需一次，以后升级时还会沿用现在的配置。感觉更精简。


Leaving a comment -- testing.


この書き方でも問題無いですが、cssは外部ファイルにしましょうか。


Okay, good.  We can read until a newline, for instance, instead of until any whitespace character.  Specifying just a character might not be what we want, of course, so we'd get a more complex interface.  This strikes me as something a compiler could optimize pretty well, which is great.

Another library design would be to return a special newline token; we could read strings in a loop until we get that sentinel value.  That design, though, doesn't lend itself to type-safety or to type genericity, which are two things the current C++ I/O Streams library was designed for.


show do milhao ANTIGO


2. `~/.gitconfig` 或 `~/.config/git/config` 文件：特定针对当前用户。


I tried it, this line is really bad, if you do a query with pageSize param eg. 1 it overrides the options and the next requests without the pageSize field will also only get 1.


should change the line comment though


I would rather err on the side of not re-discovering this particular bug.


minor - no radius here.


What version of bootstrap was this forked from? I am new to Github and do not know where to look. And what do you do when a new Bootstrap version is released? Is there a standard way to merge?


 $_SESSION['session'] = $session;

PHPの"セッション"の名前が"session"で、そこに変数$sessionを入れるという意味だと思うのですが、これだと"session"が頻出し過ぎていて後々のメンテナンス性に欠ける可能性があります。


although, if it does come back, that is evidence of a different bug, but we should test that directly.


93469234-DemandResponse2 (vidanovic) - i386-Windows-7-VisualStudio-12: OK (1685 of 1688 tests passed)

![Build Badge](http://img.shields.io/badge/build%20status-passing-green.svg) ![Test Badge](http://img.shields.io/badge/tests%20passed-99.82%25-yellow.svg)


What a great improvement with this fix.


'http://localhost/fb_study_ozawa/index.php'

のような固定値で扱う値は、defineやconstで定義するか、このくらいのコード量であれば変数でも構いません。
一度定義してから使うようにしましょう。

そのほうがコードのメンテナンス性や可読性が上がります。


你可以使用 `--global` 参数让 Git 特定读写此文件。

我觉得参考第一版，用参数可能更合适。


你当前使用仓库的 Git 目录中的 `config` 文件（就是 `.git/config`）：特指那个单独的仓库。


How important is it that this has the same model as `PatientEvent`? I don't foresee any other listeners for a long time.


このケースの !importantは不要です。 基本的に!importantは使わないものです。


再次强调，如果你使用了 `--global` 参数那么只需要运行一次，因为之后无论你在那个系统上做任何事情 ，Git 都会使用那个信息。


多分、未定義でnoticeが出たりするので、エラー回避で@を使っているかと思います。
@を使うのは結構奥の手なので、@を使わないでnoticeが出ないような構成にしてください。


Auch, you are correct. Good catch! I'll commit fix right away


如果你不设置编辑器并且你不知道 Vim 或 Emacs 是什么，当它们运行时你可能会进入一个非常困惑的状态。


div class="h1" ではなく h1要素をつかうほうが自然です。
text要素も不要です。

FB Study


@ches `:command Ag Ack`


This commit gives me the following errors.
```
Error detected while processing function <SNR>18_cpath:
line    1:
E113: Unknown option: fileignorecase
E15: Invalid expression: &fileignorecase
```

This is on a Mac with vim 7.3


Should we be renaming this if we want to take in future upstream changes from SDWebImage?


text要素を頻繁に使っていますが、普通は使いません。直接文章を記述すればOKです。


阅读第一版，感觉稍微言简意赅。全部都直译可能大家会看的比较累。

翻译辛苦了，我就提了点个人建议啊，希望不要打我，初步审校完第一遍了，我再来查查资料审第二遍。(>_<｡)


３行上の <pre> ですが、<pre> はそれなりに特殊な状況でしか使わないので、使わない方法を考えてみてください。


I took out `new FunctionIterator` because typescript complained about it being illegal to use new on a non-void constructor.

FunctionIterator() returns something (`it()`), in other words, and it doesn't allow new in that case.


<text_logout> などHTML標準に無い要素を記述しないようにしてください

HTML5リファレンス
http://www.htmq.com/html5/


:boot: :feelsgood: :repeat:


Next attempt


Whoops, guess that option is newer than I realized.


New smaller and toned down banners along with branding image for integration into top nav bar


Your "lints" are only `unconditional_recursion`, and you can get the impl relatively easily.


@UniversalMediaServer/developers what do you think about to set the parameter waitbeforestart to renderer.isTranscodeFastStart() as a default and do not put it to the appropriate renderers .conf? I have no idea how other renderers then Panasonic will manage it.


Nitpick: this should be simply "Stars" (without the colon), just like the other fields.


recommit


I think adding a link to that Openstack page would be enough. thank you!


missed `)` at the end.


Hey :wink:
So, you mean that the debug window pops up automatically?


$count
このカウントで$feedの連想配列の指定をしているかと思いますが

foreach (array_expression as $key => $value)
の形で取得出来なかったでしょうか？

http://php.net/manual/ja/control-structures.foreach.php


This can be shortened to: `Map.alter (maybe (Set.singleton uid) (Set.insert uid)) pkgname smap`

See https://hackage.haskell.org/package/containers-0.5.6.3/docs/Data-Map-Strict.html#v:alter


What is the purpose of `.remove()`?


コメントを入れると


**デモです**


nevermind. i see it now.


Try `Map.findWithDefault Set.empty` instead


Made a similar comment elsewhere… should we be modifying variable names if we intend on consuming upstream changes?


Heres a gist of the fake_data_scheduler script, available as an example for how data is written directly to InfluxDB's API

https://gist.github.com/pl12133/ea405146157bb975b3bf


no need to block, but why not first check `if isinstance(value, date)` before calling strftime? (since `datetime`s are also `date`s)


Typo here need to fix.


Edit League page's Rating Scheme field should be locked (display only) but is not yet.


Couple comments:
1. This delegate pattern for cleanup logic looks pretty solid. If we think adding a delegate callback is the best strategy, should we do it in a separate commit that could be submitted as a pull request that could be in trunk instead of maintaining it in our own fork? (I don't think the data stuff has much of a chance of making it into trunk)

2. Should we avoid modifying SDImageCache as much as possible so that we don't have to maintain divergent code changes? My hesitation in doing the delegate pattern (if it doesn't get accepted into trunk) is that we could have implemented the cleanup logic without modifying SDImageCache -  We could have disabled cleanup, and just implemented the cleanup methods in "our" code where we have the have full knowledge of MWKArticle and MWKImage - thereby removing the need to modify SDImageCache.

I realize we may still may need a private fork for NSData capabilities, but my thinking was making the least amount of changes as possible is a good strategy so we can more easily pull in upstream changes.


.\" Auto generated Nroff by NroffEdit on July 3, 2014
.pl 10.0i
.po 0
.ll 7.2i
.lt 7.2i
.nr LL 7.2i
.nr LT 7.2i
.ds LF Fang et al.
.ds RF FORMFEED[Page %]
.ds LH INTERNET DRAFT
.ds RH March 26, 2015
.ds CH MPLS-Based HSDN
.ds CF Expires <September 28, 2015>
.hy 0
.nh
.ad l
.in 0

.nf
.tl 'INTERNET-DRAFT''Luyuan Fang'
.tl 'Intended Status: Standards Track''Vijay Gill'
.tl 'Expires: September 28, 2015''Deepak Bansal'
.tl '''Microsoft'
.tl '''Fabio Chiussi'
.tl '''Cisco Systems'
.tl '''Chandra Ramachandran'
.tl '''Juniper Networks'
.tl '''Shahram Davari'
.tl 'Ebben Aries''Broadcom'
.tl 'Facebook''Linda Dunbar'
.tl 'Daniel Voyer''Huawei Technologies'
.tl 'Bell Canada''Barak Gafni'
.tl 'Wen Wang''Mellanox'
.tl 'Century Link''Andrew Qu'
.tl 'Himanshu Shah''MediaTek'
.tl 'Ciena''Jeff Tantsura'
.tl 'Ramki Krishnan''Ericsson'
.tl 'Dell''Yakov Rekhter'

.tl '''March 27, 2015'
.fi
.in 3

.ce 2
MPLS-Based Hierarchical SDN for Hyper-Scale DC/Cloud
\%draft-fang-mpls-hsdn-for-hsdc-02

.ti 0
Abstract

This document describes Hierarchical SDN (HSDN), an architectural solution to scale the Data Center (DC) and Data Center Interconnect (DCI) networks to support tens of millions of physical underlay endpoints, while efficiently handling both Equal Cost Multi Path (ECMP) load-balanced traffic and any-to-any end-to-end Traffic Engineered (TE) traffic. HSDN achieves massive scale using surprisingly small forwarding tables in the network nodes. HSDN introduces a new paradigm for both forwarding and control planes, in that all paths in the network are pre-established in the forwarding tables and the labels can identify entire paths rather than simply destinations.  The HSDN forwarding architecture is based on four main concepts: 1. Dividing the DC and DCI in a hierarchically-partitioned structure; 2. Assigning groups of Underlay Border Nodes in charge of forwarding within each partition; 3. Constructing HSDN MPLS label stacks to identify endpoints and paths according to the HSDN structure; and 4. Forwarding using the HSDN MPLS labels.

.ti 0
Status of this Memo

This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups.  Note that other groups may also distribute working documents as \%Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time.  It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

The list of current Internet-Drafts can be accessed at \%http://www.ietf.org/1id-abstracts.html

The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html

.ti 0
Copyright Notice

Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents \%(http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

.\" \# TD4  -- Set TOC depth by altering this value (TD5 = depth 5)
.\" \# TOC  -- Beginning of auto updated Table of Contents
.in 0
Table of Contents

.nf
   1. Introduction  . . . . . . . . . . . . . . . . . . . . . . . . .  4
     1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . .  6
     1.2. DC and DCI Reference Model  . . . . . . . . . . . . . . . .  8
   2. Requirements  . . . . . . . . . . . . . . . . . . . . . . . . . 10
     2.1. MPLS-Based HSDN Design Requirements . . . . . . . . . . . . 10
     2.2. Hardware Requirements . . . . . . . . . . . . . . . . . . . 11
   3. HSDN Architecture - Forwarding Plane  . . . . . . . . . . . . . 11
     3.1. Hierarchical Underlay Partitioning  . . . . . . . . . . . . 12
     3.2. Underlay Partition Border Nodes . . . . . . . . . . . . . . 14
       3.2.1. UPBN and UPBG Naming Convention . . . . . . . . . . . . 17
       3.2.2. HSDN Label Stack  . . . . . . . . . . . . . . . . . . . 17
       3.2.3. HSDN Design Example . . . . . . . . . . . . . . . . . . 18
     3.3. MPLS-Based HSDN Forwarding  . . . . . . . . . . . . . . . . 19
       3.3.1 Non-TE Traffic . . . . . . . . . . . . . . . . . . . . . 20
       3.3.1 TE Traffic . . . . . . . . . . . . . . . . . . . . . . . 22
   4. Scalability Analysis  . . . . . . . . . . . . . . . . . . . . . 23
     4.1. LFIB Sizing - ECMP  . . . . . . . . . . . . . . . . . . . . 23
     4.2. LFIB Sizing - TE  . . . . . . . . . . . . . . . . . . . . . 24
   5. HSDN Label Stack Assignment Scheme  . . . . . . . . . . . . . . 25
   6. HSDN Architecture - Control Plane . . . . . . . . . . . . . . . 27
     6.1. The SDN approach  . . . . . . . . . . . . . . . . . . . . . 27
     6.2. Distributed control plane . . . . . . . . . . . . . . . . . 27
   7.  Security Considerations  . . . . . . . . . . . . . . . . . . . 27
   8.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 28
   9.  References . . . . . . . . . . . . . . . . . . . . . . . . . . 28
     9.1  Normative References  . . . . . . . . . . . . . . . . . . . 28
     9.2  Informative References  . . . . . . . . . . . . . . . . . . 28
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 29
.fi
.in 3

.\" \# ETC -- End of auto updated Table of Contents

.bp
.ti 0
1. Introduction

.fi
With the growth in the demand for cloud services, the end-to-end cloud network, which includes Data Center (DC) and Data Center Interconnect (DCI) networks, has to scale to support millions to tens of millions of underlay network endpoints. These endpoints can be bare-metal servers, virtualized servers, or physical and virtualized network functions and appliances.

The scalability challenge is twofold: 1. Being able to scale using low-cost network nodes while achieving high resource utilization in the network; and 2. Being able to scale at low operational and computational complexity while supporting both Equal-Cost Multi-Path (ECMP) load-balanced traffic and any-to-any Traffic Engineering (TE) traffic.

Being able to scale at low cost requires to avoid the potential explosion of the routing tables in the network nodes as the number of underlay network endpoints increases. Current commodity switches have relatively small routing and forwarding tables. For example, the typical Forwarding Information Base (FIBs) and Label Forwarding Information Base (LFIBs) tables in current low-cost network nodes contain 16K or 32K entries. These small sizes are clearly insufficient to support entries for all the endpoints in the hyper-scale cloud. Address aggregation is used to ameliorate the problem, but the scalability challenges remain, since the dynamic and elastic environment in the DC/cloud often brings the need to handle finely granular prefixes in the network in order to support Virtual Machine (VM) and Virtualized Network Function (VNF) mobility.

Other factors contribute to the FIB/LFIB explosion. For example, in a typical DC using a fat Clos topology, even the support of ECMP load balancing may become an issue if the individual outgoing paths belonging to an ECMP group carry different outgoing labels, since a single destination may contribute multiple entries in the tables.

Another key scalability issue to resolve is the complexity of certain desired functions that should be supported in the network, the most prominent one being TE. Currently, any-to-any server-to-server TE in the DC/DCI is simply unfeasible, as path computation and bandwidth allocation at scale, an NP-complete problem, becomes rapidly unmanageable. Furthermore, the forwarding state needed in the network nodes for TE tunnels contributes in a major way to the explosion of the LFIBs, since each TE tunnel corresponds to an entry in the tables.

Other major scalability issues are related to the efficient creation, management, and use of tunnels, for example the configuration of protection paths for fast restoration.

Many additional scalability issues in terms of operational and computational complexity need to be resolved in order to scale the control plane and the network state. In particular, the controller-centric approach of Software Defined Networks (SDNs), which is increasingly being accepted as "the way to build the next generation clouds," still needs to be demonstrated to be scalable to the levels required in the hyper-scale DC and cloud.

Finally, the underlay network architecture should offer certain capabilities to facilitate the support of the demands of the overlay network.

In this document, we present Hierarchical SDN (HSDN), a set of solutions for all these scalability challenges in the underlay network, both in the forwarding and in the control plane.

Although HSDN can be used with any forwarding technology, including IPv4 and IPv6, it has been designed to leverage Multi Protocol Label Switching (MPLS)-based forwarding [RFC3031], using label stacks [RFC3032] constructed according to the HSDN structure. This document therefore describes MPLS-based HSDN. The HSDN underlay network is suited to support any Layer 2 or Layer 3 virtualized overlay network technology. In this document, we assume a MPLS-based overlay technology using a Virtual Network (VN) Label, which is encapsulated in the HSDN label stack. However the description can be easily generalized to any overlay technology, such as BGP/MPLS IP VPNs [RFC4364], EVPN \%[RFC7432], VXLAN [RFC7348], NVGRE \%[I-D.sridharan-virtualization-nvgre], Geneve \%[I-D.draft-gross-geneve], and other technologies.

HSDN achieves massive scale using surprisingly small LFIBs in the network nodes, while supporting both ECMP load-balanced traffic and any-to-any end-to-end TE traffic. HSDN also brings important simplifications in the control plane and in the architecture of the SDN controller.

The HSDN architecture and operation is characterized by two fundamental properties. First, all paths in the network are pre-established in the forwarding tables. Second, the HSDN labels can identify entire paths or groups of paths rather than simply destinations.

These two properties radically simplify establishing and handling tunnels. In addition to optimally handling both ECMP and Non-Equal Cost Multi Path load balancing, HSDN enables any-to-any, end-to-end, server-to-server TE at scale. With HSDN, the "cost" of establishing a tunnel is essentially eliminated, since the "tunnels" are pre-established in the network, and the TE task becomes one of path assignment and bandwidth allocation to the flows. As a larger portion of the traffic can be engineered effectively, the network can be run at a higher utilization using comparatively smaller buffers at the nodes.

The HSDN forwarding architecture in the underlay network is based on four main concepts: 1. Dividing the DC and DCI in a hierarchically-partitioned structure; 2. Assigning groups of Underlay Border Nodes in charge of forwarding within each partition; 3. Constructing HSDN MPLS label stacks to identify the end points according to the HSDN structure; and 4. Forwarding using the HSDN MPLS labels.

HSDN is designed to allow the physical decoupling of control and forwarding, and have the LFIBs configured by a controller according to a full SDN approach. The controller-centric approach is described in this document.

However, HSDN is also meant to support the traditional distributed routing and label distribution protocol approach to distribute the labels. This hybrid approach may be particularly useful during technology migration. The use of BGP Labeled Unicast (BGP-LU) for label distribution and LFIB configuration in a HSDN architecture is described in \%[I-D.fang-idr-bgplu-for-hsdn].

.ti 0
1.1. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].

.nf
Term              Definition
-----------       --------------------------------------------------
BGP               Border Gateway Protocol
BGP-LU            Border Gateway Protocol Labeled Unicast
DC                Data Center
DCGW              DC Gateway (Border Leaf)
DCI               Data Center Interconnect
DID               Destination Identifier
ECMP              Equal Cost MultiPathing
FIB               Forwarding Information Base
HSDN              Hierarchical SDN
LDP               Label Distribution Protocol
LFIB              Label Forwarding Information Base
LN                Leaf Node
MPLS              Multi-Protocol Label Switching
PID               Path Identifier
SDN               Software Defined Network
SN                Spine Node
SVR               Server
UP                Underlay Partition
UPBG              Underlay Partition Border Group
UPBG              Underlay Partition Border Node
TE                Traffic Engineering
ToR               Top-of-Rack switch
TR                Top-of-Rack switch
VN                Virtual Network
VM                Virtual Machine
VNF               Virtualized Network Function
WAN               Wide Area Network

.fi
In this document, we also use the following terms.

.in 6
.ti 3
o  End device: A physical device attached to the DC/DCI network. Examples of end devices include bare metal servers, virtualized servers, network appliances, etc.

.ti 3
o  Level: A layer in the hierarchy of underlay partitions in the HSDB architecture.

.ti 3
o  Overlay Network (ON): A virtualized network that provides Layer 2 or Layer 3 virtual network services to multiple tenants. It is implemented over the underlay network.

.ti 3
o  Path Label (PL): A label used for MPLS-based HSDN forwarding in the underlay network.

.ti 3
o  Row: A row of racks where end devices reside in a DC.

.ti 3
o  Tier: One of the layers of network nodes in a multi-layer Clos-based topology.

.ti 3
o  Underlay Network (UN): The physical network that provides the connectivity among physical end devices. It provides transport for the overlay network traffic.

.ti 3
o  Underlay Partition (UP): A logical portion of the underlay network designed according to the HSDN architecture. Underlay partitions are arranged in a hierarchy consisting of multiple levels.

.ti 3
o  VN Label (VL): A label carrying overlay network traffic. It is encapsulated in the underlay network in a stack of path labels constructed according to the HSDN forwarding scheme.

.in 3
.ti 0
1.2. DC and DCI Reference Model

Here we show the typical structure of the DC and DCI, which we use in the rest of this document to describe the HSDN architecture. We also introduce a few commonly used terms to assist in the explanation.

Figure 1 illustrates multiple DCs interconnected by the DCI/WAN.

.nf
.in 2
                   +-------------+
                   |             |
                   |     DC      |
                   |             |
                   +-------------+
                             \\-----.
                             (       ')
                         .--(.       '.---.
                        (     '      '     )
                       (       DCI/WAN      )
                        (.                .)
                         (     (        .)  \\
                          /'--' '-''---'  +-------------+
               +-------------+            |             |
               |             |            |     DC      |
               |     DC      |            |             |
               |             |            +-------------+
               +-------------+

.in 10
Figure 1. DCI\WAN interconnecting multiple DCs.

.fi
.in 3
Figure 2 below illustrates the typical structure of a Clos-based DC fabric.

.bp
.nf
.in
                                 +----+   +----+
  DCGW                 +---------+ GW +   + GW +---------+
                       |         ++--++\\ /++--++         |
                       | +--------|--|--\\--\\  |          |
                       | |        |  \\-/-\\-|--|--------+ |
                     +-+-++     +-+--+/   \\+--+-+     ++-+-+
  Spine    +---------+ SN +     | SN |     | SN +     + SN +---------+
           |         +----+\\    ++--++     ++--++\\   /+----+         |
           | +---------------\\---+  |       |  +---/---------------+ |
           | |        ...      \\    |       |    /      ...        | |
          ++-+-+ +----+  +----+ ++--++     ++--++ +----+  +----+ +-+-++
  Leaf    | LN | | LN |  | LN | | LN |     ||LN | | LN |  | LN | | LN |
          ++--++ ++--++  +---++ ++---+     +---++ ++--++  +---++ ++--++
           |   \\ /   |    |   \\ /   |       |   \\ /   |    |   \\ /   |
           |   / \\   |    |   / \\   |       |   / \\   |    |   / \\   |
           | / ... \\ |    | / ... \\ |       | / ... \\ |    | / ... \\ |
         ++++ +--+ ++++ ++++ +--+ ++++    ++++ +--+ ++++ ++++ +--+ ++++
  ToR    |TR| |TR| |TR| |TR| |TR| |TR|    |TR| |TR| |TR| |TR| |TR| |TR|
         +--+ +--+ +--+ +--+ +--+ +--+    +--+ +--+ +--+ +--+ +--+ +--+
          |    |    |    |    |    |       |    |    |    |    |    |
         +--+ +--+ +--+ +--+ +--+ +--+    +--+ +--+ +--+ +--+ +--+ +--+
  Server +--+ +--+ +--+ +--+ +--+ +--+    +--+ +--+ +--+ +--+ +--+ +--+
  rack   +--+ +--+ +--+ +--+ +--+ +--+    +--+ +--+ +--+ +--+ +--+ +--+
         +--+ +--+ +--+ +--+ +--+ +--+    +--+ +--+ +--+ +--+ +--+ +--+

.in 10
Figure 2. Typical Clos-based DC fabric topology.

.ti 10
Note: Not all nodes and links are shown in Figure 2.

.fi
.in 3
The DC fabric shown in Figure 2 uses what is known as a spine and leaf architecture with a multi-stage Clos-based topology interconnecting multiple tiers of network nodes. The DC Gateways (DCGWs) connect the DC to the DCI/WAN. The DCGW connect to the Spine Nodes (SNs), which in turn connect to the Leaf Nodes (LFs). The Leaf Nodes connect to the Top-of-Rack switches (ToRs). Each ToR typically resides in a rack (hence the name) accommodating a number of servers connected to their respective ToR. The servers may be bare metal or virtualized.

Each tier of switches and the connectivity between switches is designed to offer a desired capacity and provide sufficient bandwidth to the servers and end devices.

Figure 2 is not meant to represent the precise topology of the DC. In fact, the precise topology and connectivity between the tiers of switches depends on the specific design of the DC. More or less tiers of switches (spines or leaves) or asymmetric topologies, not shown in the figure, may be used. A precise description of the possible topologies and related design criteria is out of the scope of this document.

What is relevant for this document is the fact that a typical large-scale DC topology does not have all the tiers fully connected to the adjacent tier (i.e., not all network nodes in a tier are necessarily connected to all network nodes in the adjacent tiers). This is especially true for the tiers closer to the endpoints, and is due to the sheer number of connections and devices (in other words, in a large, fat Clos there are too many network nodes in some tiers for all network nodes to connect to one another), and to the physical constraints of the DC (i.e., the network nodes may be located physically apart in separate rooms or buildings, and full connectivity may become too costly).

The actual connectivity is typically organized following an aggregation/multiplexing connectivity architecture that consolidates traffic from the edges into the leafs and spines, while allowing for over-subscription in order to strike a reasonable trade-off between cost and available capacity. The connectivity between each tier uses some form of shuffle-exchange topology that attempts to "mix" the available paths while taking in account the physical constraints.

The key observation is that it is impractical, uneconomical, and ultimately unnecessary to use a fully connected Clos-based topology in a large scale DC. Because of the physical constraints, the topology of a large DC is not a flat, fully-connected, multi-layer Clos, but rather has some form of hierarchy. The HSDN architecture recognizes this fact, and uses it to dramatically simplify forwarding and control planes using an approach that is also hierarchical.

.ti 0
2. Requirements

.ti 0
2.1. MPLS-Based HSDN Design Requirements

The following are the key design requirements for MPLS-based HSDN solutions.

.in 6
.ti 3
1) MUST support millions to tens of millions of underlay network endpoints in the DC/DCI.

.ti 3
2) MUST use very small LFIB sizes (e.g., 16K or 32K LFIB entries) in all network nodes.

.ti 3
3) MUST support both ECMP load-balanced traffic and any-to-any, end-to-end, server-to-server TE traffic.

.ti 3
4) MUST support ECMP traffic load balancing using a single forwarding entry in the LFIBs per ECMP group.

.ti 3
5) MUST require IP lookup only at the network edges.

.ti 3
6) MUST support encapsulation of overlay network traffic, and support any network virtualization overlay technology.

.ti 3
7) MUST support control plane using both full SDN controller approach, and traditional distributed control plane approach using any label distribution protocols.

.in 3
.ti 0
2.2. Hardware Requirements

The following are the hardware requirements to support HSDN.

.in 6
.ti 3
1) The server NICs MUST be able to push a HSDN label stack consisting of as many path labels as levels in the HSDN hierarchical partition (e.g., 3 path labels).

.ti 3
2) The network nodes MUST support MPLS forwarding.

.ti 3
3) The network nodes MUST be able perform ECMP load balancing on packets carrying a label stack consisting of as many path labels as levels in the HSDN hierarchical partition, plus one or more VN label/header for the overlay network (e.g., 3 path labels + 1 VN label/header).

.in 3
.ti 0
3. HSDN Architecture - Forwarding Plane

As mentioned above, a primary design requirement for HSDN is to enable scalability of the forwarding plane to tens of millions of network endpoints using very small LFIB sizes in all network nodes in the DC/DCI, while supporting both ECMP and any-to-any server-to-server TE traffic.

The driving principle of the HSDN forwarding plane is "divide and conquer" by partitioning the forwarding task into local and independent forwarding. When designed properly, such an approach enables extreme horizontal scaling of the DC/DCI.

HSDN is based on four concepts:

.in 6
.ti 3
1) Dividing the underlay network in a hierarchy of partitions;
.ti 3
2) Assigning groups of Underlay Partition Border Nodes (UPBN) to each partition, in charge of forwarding within the corresponding partition;
.ti 3
3) Constructing HSDN label stacks for the endpoint Forward Equivalency Classes (FECs) in accordance with the underlay network partition hierarchy;
.ti 3
4) Configuring the LFIBs in all network nodes and forwarding using the label stacks.

.in 3
As explained below, the HSDN label stacks can be used to identify entire paths to each endpoint, rather than simply the destination endpoint itself. As a matter of fact, the HSDN solution is meant to be configured with all possible paths in the network pre-established in the LFIBs in the network nodes. In this case, a FEC per path to each endpoint is defined. However, because of the way the HSDN architecture is designed, the required local number of entries in the LFIB of each network node remains surprisingly small.

In this section, we explain in detail each of these concepts. Scalability analysis for both ECMP load-balanced and TE traffic is presented in Section 4. In Section 5, we describe a possible label stack assignment scheme for HSDN.

.ti 0
3.1. Hierarchical Underlay Partitioning

HSDN is based on dividing the DC/DCI underlay network into logical partitions arranged in a multi-level hierarchy.

The HSDN hierarchical partitioning is illustrated in Figure 3.

.bp
.nf
.in
           +----------------------------------------------------------+
  UP       |                          UP0                             |
  Level 0  |                                                          |
           |+---------------------------------+   +------------------+|
           ||                                 |...|                  ||
           +|---------------------------------|---|------------------|+
  UP        |             UP1-0               |   |       UP1-N      |
  Level 1   |                                 |   |                  |
            |+-------------+   +-------------+|   |   +-------------+|
            ||             |...|             ||   |...|             ||
            +|-------------|---|-------------|+   +---|-------------|+
  UP         |   UP2-0-0   |   |   UP2-0-N   |        |  UP2-N-N    |
  Level 2    |             |   |             |        |             |
             +-----+ +-----+   +-----+ +-----+        +-----+ +-----+
             | SVR | | SVR |   | SVR | | SVR |        | SVR | | SVR |
             |-----|-|-----|   |-----|-|-----|        |-----|-|-----|
  Overlay    |VM|VM| |VM|VM|   |VM|VM| |     |        |VM|VM| |     |
  Level      |-----| |-----|   |-----| |     |        |-----| |     |
             |VM|VM| |VM|VM|   |VM|VM| |     |        |VM|VM| |     |
             +-----+ +-----+   +-----+ +-----+        +-----+ +-----+

.in 3
Figure 3. HSDN underlay network hierarchical partitioning of DC/DCI.

.fi
.in 3
The hierarchy consists of multiple levels of Underlay Partitions (UPs). For simplicity, we describe HSDN using three levels of partitioning, but more or less levels can be used, depending on the size and architecture of the overall network, using similar design principles (as shown below, three levels of partitions are sufficient to achieve scalability to tens of millions servers using very small LFIBs).

The levels of partitions are nested into a hierarchical structure. At each level, the combination of all partitions covers the entire DC/DCI topology. In general, within each level, the UPs do not overlap, although there may be design scenarios in which overlapping UPs within a level may be used. The top level (Level 0) consists of a single underlay partition UP0 (the HSDN concept can be extended to multi-partitioned Level 0).

We use the following naming convention for the UPs:

.in 6
.ti 3
-  Partitions at Level i are referred to as UPi (e.g., UP0 for Level 0, UP1 for Level 1, UP2 for Level2, and so on).

.ti 3
-  Within each level, partitions are identified by a rightmost sequential number (starting from 1) referring to the corresponding level and a set of sequential number(s) for each partition in a higher level that the specific partition is nested into.

For example, at Level 1, there are N partitions, referred to as UP1-1 to UP1-N.

Similarly, at Level 2, there are M partitions for each Level 1 partitions, for a total of NxM partitions. For example, the Level 2 partitions nested into Level 1 partition UP1-1 are UP2-1-1 to UP2-1-M, while the ones nested into UP1-N are UP2-N-1 to UP2-N-M.

.ti 3
-  Note that for simplicity in illustrating the partitioning, we assume a symmetrical arrangement of the partitions, where the number of partitions nested into each partition at a higher level is the same (e.g., all UP1 partitions have M UP2 partitions). In practice, this is rarely the case, and the naming convention can be adapted accordingly for different numbers of partitions nesting into each higher level partition (e.g., partition UP1-1 has M1 UP2 partitions, partition UP1-2 has M2 UP2 partitions, and so on).

.in 3
The following considerations complete the description of Figure 3.

.in 6
.ti 3
o  The servers (bare metal or virtualized) are attached to the bottom UP level (in our case, Level 2). A similar naming convention as the one used for the partitions may be used.

.ti 3
o  In Figure 3, we also show an additional Overlay Level. This corresponds to the virtualized overlay network (if any) providing Virtual Networks (VN) connecting Virtual Machines (VMs) and other overlay network endpoints. Overlay network traffic is encapsulated by the HSDN underlay network. As mentioned in the Introduction, the HSDN underlay network is suited to support any Layer 2 or Layer 3 virtualized overlay network technology, such as BGP/MPLS IP VPNs [RFC4364], EVPN \%[RFC7432], VXLAN [RFC7348], NVGRE \%[I-D.sridharan-virtualization-nvgre], Geneve \%[I-D.draft-gross-geneve], and other technologies. A full description of the encapsulation of these technologies into the HSDN underlay label stack is out of scope of this document and will be addressed in a separate document.

.in 3
The UPs are designed to contain one or more tiers of switches in the DC topology or nodes in the DCI. The key design criteria in defining the partitions at each layer is that they need to follow the "natural" connectivity implemented in the DC/DCI topology. An example is given below to further clarify how the partitions are designed.

.ti 0
3.2. Underlay Partition Border Nodes

Once the HSDN hierarchical partitioning is defined, Underlay Partition Border Nodes (UPBNs) are assigned to each UP. This is illustrated in Figure 4.

.nf
.in
  +++++++++++++++++++++++++++++++++++++++++++++++++
  |                     UP0              <--------|-----+
  |                                               |     |
  | +++++++++++++++++++++++++++++++++++++++++++++ |     |
  | |   +-----------------------------------+   | |     |
  | |   | +---------+  UPBG1-i  +---------+ |   | |     |
  | |   | | UPBN1-i |    ...    | UPBN1-i | |   | |     |
  | |   | +---------+           +---------+ |   | |     |
  | |   +-----------------------------------+   | |     |
  ++|+++++++++++++++++++++++++++++++++++++++++++|++     |
    |                   UP1              <------|----+  |   +---------+
    | +++++++++++++++++++++++++++++++++++++++++ |    |  +---|   PL0   |
    | | +-----------------------------------+ | |    |      +---------+
    | | | +---------+ UPBG2-i-j +---------+ | | |    +------|   PL1   |
    | | | |UPBN2-i-j|    ...    |UPBN2-i-j| | | |           +---------+
    | | | +---------+           +---------+ | | |    +------|   PL2   |
    | | +-----------------------------------+ | |    |      +---------+
    ++|+++++++++++++++++++++++++++++++++++++++|++    |  +---|   VL    |
      |                 UP2              <----|------+  |   +---------+
      |          +---------------+            |         |
      |          |   SVR-i-j-k   |            |         |
      |          | +-----------+ |            |         |
      |          | |    NVE    |<-------------|---------+
      |          | +-----+-----+ |            |
      +++++++++++| | VM  | VM  | |+++++++++++++
                 | +-----+-----+ |
                 | | VM  | VM  | |
                 +-+-----+-----+-+

.in 10
Figure 4. UBPNs, UBPGs, and label stack assignment.

.fi
.in 3
The UPBNs serve as the connecting nodes between adjacent partitions. As such, the UPBNs belong to two partitions in adjacent levels in the hierarchy and they constitute the entry points for traffic from the higher level partition destined to the corresponding lower level partition (and vice-versa, they are the exit points for traffic from a lower level partition to a higher level partition). As such, they constitute the forwarding end destinations within each partition.

In order to provide sufficient capacity and support traffic load balancing between the levels in the hierarchy, multiple UPBNs are assigned to each partition. The UPBNs for each partition are grouped into an Underlay Partition Border Group (UPBG). As shown below, using an appropriate Label Stack Assignment scheme all UPBNs in a UPBG can be made identical for ECMP traffic forwarding (i.e., the ECMP entries in the LFIBs in all UPBNs in a UPBG are identical). Thus, for ECMP traffic load balancing, all UPBNs belong to the same FEC as far as the higher level partition is concerned. For TE traffic, a desired UPBN within a UPBG group may need to be specified, and thus the UPBNs in a UPBG are not forwarding-wise equivalent.

In practice, the UPs are designed by finding the most advantageous way to partition the DC Clos-based topology and the DCI topology. As mentioned above, the connectivity of any large-scale DC is not fully flat, but rather contains some sort of hierarchical organization. Recognizing the hierarchy of the physical connectivity is an important starting point in the design of the partitions.

Within the DC, the UPBNs in each level are subsets of the network nodes in one of the tiers that form the multi-stage Clos architecture.

In general, in addition to the UPBNs, the UPs may internally contain tiers of network nodes that are not UPBNs. A specific design example to further illustrate the HSDN partitioning is provided below.

As explained in more detail below, for forwarding purposes, by partitioning the DC/DCI in this manner and using HSDN forwarding, the UPBNs need to have entries in their LFIBs only to reach destinations in the two partitions to which they belong to (i.e., their own corresponding lower-level partition and the higher-layer partition to which they nested to). The network nodes inside the UPs only need to have entries in their LFIB to reach the destinations in their partition.

Similarly, in order to establish all possible paths in the entire network, the UPBNs need to have entries in their LFIBs only for all possible paths to the destinations in the two partitions to which they belong to.

From these considerations, a first design heuristic for choosing the partitioning structure is to keep the number of partitions nested at each level into the higher level relatively small for all levels. For the lowest level, the number of endpoints (servers) in each partition should also be kept to manageable levels.

Clearly, the design tradeoff is between the size and the number of partitions at each level. Although finding the optimal design choice may require a little trial-and-error computation of different options,fortunately, for most practical deployments, it is relatively simple to find a good tradeoff that achieves the desired scalability to millions or tens of millions of endpoints.

.ti 0
3.2.1. UPBN and UPBG Naming Convention

We use a similar naming convention for the UPBNs and UPBGs as the one used for the UPs:

.in 6
.ti 3
-  UPBNi is a UPBN between partitions at Level(i) and Level(i-1). Similarly for UPBG.

.ti 3
-  Within each level, the UPBNs are identified by a set of sequential number(s) equal to the corresponding sequential number(s) of the corresponding partition within that level.

For example, at Level 1, UPBN1-1 corresponds to partition UP1-1, and connects UP0 with UP1-1. UPBN1-N corresponds to partition UP1-N and connects UP0 with UP1-N, and so on. Similarly for UPBG.

At Level 2, UPBN2-1-1 corresponds to partition UP2-1-1 and connects UP1-1 with UP2-1-1, and so on. Similarly for UPBG.

Note that the UPBNs within an UPBGs can be further distinguished using an appropriate naming convention (for example, using an additional sequential number within the UPBG), which for simplicity is not shown here. This more granular naming convention is needed to configure the paths and the TE tunnels.

.in 3
.ti 0
3.2.2. HSDN Label Stack

In MPLS-Based HSDN, an MPLS label stack is defined and used for forwarding. The key notion in HSDN is that the label stack is defined and the labels are assigned in accordance with the hierarchical partitioned structure defined above.

The label stack, shown in Figure 4 above, is constructed as follows.

.in 6
.ti 3
-  The label stack contains as many Path Labels (PLs) as levels in the partitioning hierarchy.

.ti 3
-  Each PL in the label stack is associated to a corresponding level in the partition hierarchy and is used for forwarding at that level.

In the scenario of Figure 4, PL0 is associated to Level 0 and is used to forward to destinations in UP0, PL1 is associated to Level 1 and is used to forward to destinations in any UP1 partitions, and PL2 is associated to Level 2 and is used to forward to destinations in any UP2 partitions.

.ti 3
-  A VN Label (VL) is also shown in the label stack in Figure 4. This label is associated to the Overlay Level and is used to forward in the overlay network. The VL is simply encapsulated in the label stack and transported in the HSDN underlay network. As mentioned above, the HSDN underlay network is suited to support any Layer 2 or Layer 3 virtualized overlay network technology, and thus the VL may be a label, a tag, or some other identifier, depending on the overlay technology used. The details of the VL encapsulation and processing for different overlay technologies are out of scope of this document.

.in 3
Each endpoint in the DC/DCI is identified by a corresponding label stack. For a given endpoint, the label stack is constructed in such a way that the PLO specifies the UP1 to which the endpoint is attached to, the PL1 specifies the UP2 to which the endpoint is attached to, and the PL2 specifies the FEC in the UP2 corresponding to the endpoint.

.in 3
The labels in the HSDN label stack can identify entire paths, rather than simply the end destination within the corresponding partition. This can be used to bring dramatic simplifications in handling tunnels and TE traffic in particular, as further explained below.

A scheme to assign the PL labels in the HSDN label stack is described below.

.ti 0
3.2.3. HSDN Design Example

We use an example to further explain the HSDN design criteria to define the hierarchically-partitioned structure of the DC/DCI. We use the same design example in the Scalability Analysis section below to show the LFIB sizing with ECMP and TE traffic.

To summarize some of the design heuristics for the HSDN underlay partitions:

.in 6
.ti 3
-  The UPs should be designed to follow the "natural" connectivity topology in the DC/DCI.

.ti 3
-  The number of partitions at each level nested into the higher level should be relatively small (since they are FEC entries in the LFIBs in the network nodes in the corresponding levels).

.ti 3
-  The number of endpoints (servers) in each partition in the lowest level should be relatively small (since they are FEC entries in the LFIBs in the network nodes in the lowest level).

.ti 3
-  The number of levels should be kept small (since it corresponds to the number of path labels in the stack).

.ti 3
-  The number of tiers in each partition in each level should be kept small. This is due to the multiplicative fanout effect for TE traffic (explained below), which has a major impact on the LFIB size needed to support any-to-any server-to-server TE.

.in 3
The HSDN forwarding plane design consists in finding the best tradeoff among these conflicting objectives. Although the optimal design choices ultimately depend on the specific deployment, fortunately, it is generally rather straightforward to identify design choices that can support scalability to millions or tens of millions of servers.

Here we describe a design example to illustrate that a three-level HSDN hierarchy is sufficient to scale the DC/DCI to tens of millions of servers.

With three levels, a possible design choice for the UP1s is to have each UP1 correspond to a DC. With this choice, the UP0 corresponds to the DCI and the UPBN1s are the DCGWs in each DC (the UPBG1s group the DCGWs in each DC).

Once the UP1s are chosen this way, a possible design choice for the UP2s is to have each UP2 correspond to a group of racks, where each group of racks may correspond to a portion of a row of racks, an entire row of racks, or multiple rows of racks. The specific best choice of how many racks should be in a group of racks corresponding to each UP2 ultimately depends on the specific connectivity in the DC and the number of servers per racks.

While precise numbers depend on the specific technologies used in each deployment, here and in the Scalability Analysis section below we want to give some ideas of the scaling capabilities of HSDN. For this purpose, we use some hypothetical yet reasonable numbers to characterize the partitioning design example.

Assume the following: a) 20 DCs connected via the DCI/WAN; b) 50 servers per rack; c) 20 racks per group of racks; d) 50 groups of racks per DC.

With these numbers, there are 500K servers per DC, for a total of 10M underlay network endpoints in the DC/DCI.

In the HSDN structure in this example, there are 20 UP1s, 500 UP2s per UP1, and 1000 servers per UP2.

.ti 0
3.3. MPLS-Based HSDN Forwarding

The hierarchically partitioned structure and the corresponding label stack are used in HSDN to scale the forwarding plane horizontally while using LFIBs of surprising small sizes in the network nodes.

As explained above, each label in the HSDN label stack is associated with one of the levels in the hierarchy and is used to forward to destinations in the underlay partitions at that level.

With HSDN, by superimposing a hierarchically-partitioned structure and using a label stack constructed according to such a structure, we are able to impose a forwarding scheme that is aggregated by construction. This translates in dramatic reductions in the size of the LFIBs in the network nodes, since each node only needs to know a limited portion of the forwarding space.

HSDN supports any label assignment scheme to generate the labels in the label stack. However, if a label assignment scheme that is consistent with the HSDN structure is used, additional simplifications of the LFIBs and the control plane can be achieved.

In Section 5 below, we present one example of such a scheme, where the labels in the label stack represent the "physical" location of the endpoint, expressed according to the HSDN structure. For TE traffic, the labels represent a specific path towards the desired destination through the HSDN structure.

In the Scalability Analysis section and in the Control Plane section below we assume that such a Label Assignment scheme is used.

In the rest of this section, we describe the life of a packet in the HSDN DC/DCI. We use the specific design example described in Section 3.2.3 above to help in the explanation, but of course the forwarding would be similar for other design choices.

.ti 0
3.3.1 Non-TE Traffic

We first describe the behavior for ECMP load-balanced, non-TE traffic. In the HSDN DC/DCI, for a packet that needs to be forwarded to a specific endpoint in the underlay network, the outer label PL0 specifies which UP1 contains the endpoint. Let's refer to this UP1 as UP1-a. For ECMP traffic, the PL0 binding is with a FEC corresponding to the UPBG1-a associated with UP1-a. Note that all the endpoints reachable via UP1-a are forwarded using the same FEC entry for Level 0 in the hierarchical partitioning.

Once the packet reaches one of the network nodes UPBN1-a in the UPBG1-a group (the upstream network nodes perform ECMP load balancing, thus the packet may enter UP1-a via any of the UPBN1-a nodes), the PL0 is popped and the PL1 is used for forwarding in the UP1-a (to be precise, because of penultimate hop popping, it is the network node immediately upstream of the chosen UPBN1-a that pops the label P0).

The PL1 is used within UP1-a to reach the UP2 which contains the endpoint. Let's refer to this UP2 as UP2-a. In the UP2 network nodes the PL1 binding is with a FEC corresponding to the UPBG2-a associated with UP2-a. Similarly as above, note that all the endpoints reachable via UP2-a are forwarded using the same FEC entry for Level 1 in the hierarchical partitioning.

Once the packet reaches one of the network nodes UPBN2-a in the UPBG2-a group (once again, the upstream network nodes perform ECMP load balancing, so the packet may transit to any of the UPBN2-a nodes), the PL1 is popped and the PL2 is used for the rest of the forwarding (again, to be precise, the penultimate network node upstream of UPBN2-a is the one popping the PL1 label).

The PL2 is used within UP2-a to reach the desired endpoint. Note that the UPBN2 nodes and the network nodes in the UP2s have entries in their LFIBs only to reach endpoints within their UP2. They can reach endpoints in other UP2s by using a FEC entry corresponding to the UP2 containing the destination endpoint, identified by PL1.

The following two observations help in further clarifying the forwarding operation above.

.in 6
.ti 3
-  The PL0 is used for forwarding from the source to the UPBN1-a. For a packet originating from an endpoint attached to a certain UP2, say UP2-b, nested to a different UP1, say UP1-b, PL0 is used for forwarding in all network nodes that the packet transits until it reaches the UPBN1-a. This includes network nodes in UP2-b and UP1-b (i.e., "on the way out" from UP2). It also includes one of the UPBN1-b nodes.

It is important to note, however, that the PL0 is not popped at the UPBN1-b, since it is used for forwarding to the destination UPBN1-a.

.ti 3
-  It should be pointed out that, in order to achieve route optimization, not all packets need to use the entire HSDN label stack and thus not all packets carry a three-label MPLS stack. For example, a packet originating from the endpoint in UP2-b and destined to an endpoint in the same UP2-b should not be forwarded all the way to the highest level in the hierarchy and back, but should be forwarded to the desired endpoint at the first node in the UP2 that contains an entry to that desired endpoint. In other words, the packet only needs PL2 to be properly forwarded, since it should never "go out" of UP2.

To optimize the label assignment and forwarding scheme, a single "turn around" label value can be used to identify traffic that needs to be kept within the partition once the corresponding UPBN is reached. For example, a packet destined to an endpoint in the same UP2-b of the originating server that needs to only reach the UPBN2-b, may carry a PL1 corresponding to the "turn around" label value and a PL2 corresponding to the desired endpoint within UP2, and does not need a PL0.

Similarly, a packet originating from the endpoint in UP2-b and destined to an endpoint in a different UP2 nested in the same UP1-b only needs PL1 and PL2 to be forwarded.

.in 3
In the case of ECMP load-balanced non-TE traffic, the labels in the HSDN label stack identify ECMP groups for each destination in the corresponding partition. In this way, at each node in the partition, the outgoing label is the same for all paths belonging to the same ECMP group. A label allocation scheme for this is described below.

.ti 0
3.3.1 TE Traffic

Handling TE traffic in the hyper-scale DC/DCI presents major scalability challenges, since each TE tunnel contributes one entry in the forwarding tables, and the TE path and bandwidth allocation computation is a NP-complete problem.

HSDN introduces radical simplifications in establishing and handling tunnels, and in supporting TE in particular.

In HSDN, all paths in the network can be pre-established in the LFIBs. Because of the way the HSDN architecture is constructed, the number of entries that have to be stored in the local LFIB in each network node remains surprisingly small.

In this case, the labels in the HSDN stack identify entire paths, or groups of paths, to each destination in each partition, rather than just the destination itself.

With HSDN, since the "cost" of establishing a tunnel is essentially eliminated (all "tunnels" are pre-established in the network), and the TE task becomes one of path assignment and bandwidth allocation to the flows. Furthermore, the hierarchical structure of HSDN makes it possible to devise algorithms and heuristics for path and bandwidth allocation computation that operate largely independently in each partition, and are therefore computationally feasible even at large scale. A description of such algorithms is out of scope of this document. As a larger portion of the traffic can be engineered effectively, the network can be run at a higher utilization using comparatively smaller buffers at the nodes.

Since all paths can be accommodated in the LFIBs, HSDN makes it possible to support "TE Max Case" with small LFIB sizes. In
TE Max Case, all sources are connected to all destinations (e.g.,
server to server) with TE tunnels, the tunnels using all possible
distinct paths in the network. TE Max Case gives therefore an upper
bound to the number of TE tunnels (and consequently, LFIB entries) in
the network.

The fact that the LFIBs remain relatively small even when all possible paths are configured is the consequence of two desirable properties of HSDN.

First, since in HSDN the individual UPs are designed in such a way to be relatively small, the number of paths in each partition can be kept
to a manageable number.

Second, the hierarchical structure of HSDN makes it possible to use the partitioning astutely to break the "TE Fanout Multiplicative Effect," which defines the number of paths to a destination, and can easily contribute to the LFIB explosion as the number of hops and the fanout of each hop to each destination in the network increases. As explained below, with the hierarchical structure, the TE Fanout Multiplicative Effect is only multiplicative within each level in the hierarchy. Thus, by properly designing the partitioning, the multiplicative effect can be kept to a manageable level.

In the case of TE traffic, the processing of the different labels in the label stack is similar to what described above for ECMP load-balanced non-TE traffic. However, the labels are bound to FECs identifying a specific path within each UPs that is traversed.

.ti 0
4. Scalability Analysis

In this section, we compute the maximum size of the LFIBs for non-TE/ECMP traffic and any-to-any server-to-server TE traffic.

.ti 0
4.1. LFIB Sizing - ECMP

For ECMP traffic, at each level, all destinations belonging to the same partition at a lower level are forwarded using the same FEC entry in the LFIB, which identifies the destination UPBG for that level, or the destination endpoint at the lower level. Since the UPs are designed in such a way to keep the number of destinations small in all UPs, and the network nodes only need to know how to reach destinations in their own UP and in the adjacent UP at the higher level in the hierarchy, this translate to the fact that hyper scale of the DC/DCI can be achieved with very small LFIB sizes in all the individual network nodes.

The worst case for the LFIB size occurs at one of the network nodes that serve as UPBNs for one of the levels of UPs in the hierarchy. The level where the LFIB size occurs depend on the specific choice of the partitioning design.

.ti 0
4.2. LFIB Sizing - TE

As noted above, TE traffic may add a considerable number of entries to LFIB, since it creates one new FEC per TE tunnel to each destination.

HSDN provides a solution to this problem, and in fact, HSDN can support any-to-any server-to-server "TE Max Case" with small LFIB sizes.

In a Clos Topology (the analysis can be extended to generic topologies), the number of paths in a UP with N destination can be easily computed. The number of paths (and the maximum number of LFIB entries) is equal to the products of the switch fanout in each tier traversed from the source to the destination in that UP.
This is the "TE Fanout Multiplicative Effect" mentioned above, which is illustrated in Figure 5. Accordingly:

Total # LFIB Entries for TE Max Case = N * F1 * F2 * ... * F(M-1)

Where Fi is the fanout of a switch in each tier traversed to the destination, M is the number of tiers in the UP, and N is the number of destinations in the UP.

Once again, by properly designing the UPs, the TE Fanout Multiplicative Effect can be kept under control, since the path computation is local for each of the UPs. HSDN breaks the multiplicative effect, since the TE Fanout Multiplicative Effect is multiplicative only within each UP, rather than in the entire network, and the "multiplication" restarts at each level of the hierarchy.

.bp
.nf
.in 8
                          +-------+
   Source                 |  Src  |
                          | Node  |
                          +-------+  F1=3
                         /    |    \\
                        /     |     \\
                +-------+ +-------+ +-------+
   Tier 1       |       | |       | |       |
                |  Node | |  Node | |  Node |
                +-+-----+ +---+---+ +-----+-+  F2=3
                  |   \\ \\ /   |   \\ / /   |
                  |    \\/ \\---|---/ \\/    |
                  |   / \\ /---|---\\ / \\   |
                  | /  / \\    |    / \\  \\ |
                +-+-----+ +---+---+ +-----+-+
   Tier 2       |       | |       | |       |
                |  Node | |  Node | |  Node |
                +-------+ +-------+ +-------+  F3=2
                  \\      \\ /     \\ /     /
                   \\     / \\     / \\    /
                    \\   /    / \\    \\  /
                     +-------+ +-------+
                     |       | |       |
   Tier 3            | Node  | | Node  |
                     +-------+ +-------+  F4=1
                         \\        /
                          \\      /
                          +-------+
                          | Dest  |
   Destination            | Node  |
                          +-------+

.in 12
Figure 5. Fan out multiplicative effect with TE.

.fi
.ti 0
5. HSDN Label Stack Assignment Scheme

.in 3
HSDN can use any scheme to assign the labels in the label stack. However, if a label assignment scheme which assigns labels in a way consistent with the HSDN structure, important simplifications can be achieved in the control plane and in the LFIBs.

For non-TE FECs, the HSDN label assignment scheme assigns labels according to the "physical" location of the endpoint in the HSDN structure. Continuing our design example from above, for an endpoint X in UP2-a, PL0 would identify the DC in which the endpoint is located, PL1 would identify the group of racks in which the endpoint is located within the DC, and PL2 would identify the endpoint within the group of servers within the DC.

For TE FECs, the HSDN label assignment scheme assigns labels to identify a specific path in each UP that is traversed. In our example, for a specific TE tunnel to endpoint X, PL0 would identify the specific path that should be followed in the DCI, PL1 would identify the path that should be followed within the DC to reach the group of racks, and PL2 would identify the path to reach the endpoint within the group of racks (if there are multiple paths).

In order to assign labels to both non-TE traffic and TE traffic, HSDN uses a label format in which the labels are divided into two logical sub-fields, one identifying the destination within the UP, called Destination Identifier (DID), and one identifying the path, called Path Identifier (PID). The Path Identifier is only relevant for TE traffic, and can be zero for non-TE traffic. The HSDN Label format is illustrated in Figure 6.

.nf
.in 5
      0                          d                       19
      +++++++++++++++++++++++++++++++++++++++++++++++++++++
      |  Destination Identifier  |     Path Identifier    |
      +++++++++++++++++++++++++++++++++++++++++++++++++++++
      |<-------- d bits -------->|<-----(20-d bits)------>|
      LSB                                               MSB

.in 20
Figure 6. HSDN Label format.

.fi
.in 3
Depending on the LFIB configuration, the two MSBs may be reserved for identifying the layer (i.e., whether the label is PL0, PL1, or PL2) to resolve ambiguity (not shown in Figure 6). Note, however, that this is not strictly necessary and the same function of identifying the layer can be achieved by simply allocating "turn around" entries in the nodes, so an individual node always sees the same label in the stack.

By properly designing the UPs, this label assignment scheme can support the desired scalability and the support of end-to-end TE traffic.

Note that by using this type of label assignment scheme important benefits can be achieved, including:

.in 6
.ti 3
-  The LFIBs become rather "static," since the FECs are tied to "physical" locations and paths, which change infrequently. This simplifies the use of the SDN approach to configure the LFIBs via a controller.

.ti 3
-  All paths in each ECMP group use the same outgoing labels. This guarantees that a single LFIB entry can be used for each ECMP group.

.in 3
The label stack needs to be imposed at the entry points. For an endpoint, this implies that the server NIC must be able to push a three-label stack of path labels (in addition to possibly push one additional VL label for the overlay network).

.ti 0
6. HSDN Architecture - Control Plane

HSDN has been designed to support the controller-centric SDN approach in a scalable fashion. HSDN also supports the traditional distributed control plane approach.

HSDN introduces important simplifications in the control plane and in the network state as well.

.ti 0
6.1. The SDN approach

In the controller-centric SDN approach, the SDN controller configures the LFIBs in all the network nodes. With HSDN, the hierarchical partitioned structure offers a natural framework for a distributed implementation of the SDN controller, since the control plane in each UP is largely independent from other UPs.

For example, a possible architecture uses a SDN controller for each UP. Such SDN partition controller is in charge of configuring the LFIBs in the network nodes in the corresponding UP.

The SDN partition controller may also be in charge of TE computation. With proper design of the UPs, TE path computation algorithms which perform partition-local computation while approach global optimality can be used.

.ti 0
6.2. Distributed control plane

HSDN can also use the traditional distributed routing protocol approach to distribute HSDN labels. An example using BGP-LU [RFC3107] is presented in [I-D.fang-idr-bgplu-for-hsdn].

.ti 0
7.  Security Considerations

When the SDN approach is used, the protocols used to configure the LFIBs in the network nodes MUST be mutually authenticated.

For general MPLS/GMPLS security considerations, refer to [RFC5920].

Given the potentially very large scale and the dynamic nature in the cloud/DC environment, the choice of key management mechanisms need to be further studied.

To be completed.

.ti 0
8.  IANA Considerations

TBD.

.ti 0
9.  References

.ti 0
9.1  Normative References

.in 14
.\" \# REF  -- Managed reference list. Syntax: [RFC]nnnn[;Label], ... ,[RFC]nnnn[;Label]
.\" \# 2119, 3031, 3032, 3107, 4364
.\" \# ERF -- End of managed reference list
.ti 3
[RFC7432]  Sajassi et al., "BGP MPLS Based Ethernet VPN", RFC 7432, February 2015.

.ti 0
9.2  Informative References

.in 14
.\" \# REF  -- Managed reference list. Syntax: [RFC]nnnn[;Label], ... ,[RFC]nnnn[;Label]
.\" \# 5920
.\" \# ERF -- End of managed reference list
.ti 3
[RFC7348]  M. Mahalingam et al., "Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks",  RFC 7348, August 2014.

.ti 3
[I-D.fang-idr-bgplu-for-hsdn]  L. Fang et. al., "BGP-LU for HSDN Label Distribution", draft-fang-idr-bgplu-for-hsdn-00 (work in progress), March 2015.

.ti 3
[I-D.sridharan-virtualization-nvgre] M. Sridharan et al., "NVGRE: Network Virtualization using Generic Routing Encapsulation", draft-sridharan-virtualization-nvgre-07  (work in progress), November 2014.

.ti 3
[I-D.draft-gross-geneve] J. Gross et al., "Geneve: Generic Network Virtualization Encapsulation", draft-gross-geneve-02 (work in progress), October 2014.

.ti 0
Authors' Addresses

.nf
.in 3
Luyuan Fang
Microsoft
5600 148th Ave NE
Redmond, WA 98052
Email: lufang@microsoft.com

Vijay Gill
Microsoft
5600 148th Ave NE
Redmond, WA 98052
Email: vgill@microsoft.com

Deepak Bansal
Microsoft
15590 NE 31st St.
Redmond, WA 98052
Email: dbansal@microsoft.com

Fabio Chiussi
Cisco Systems
500 108th Avenue N.E., Suite 500
Bellevue, WA 98004
Email: fchiussi@cisco.com

Chandra Ramachandran
Juniper Networks
Electra, Exora Business Park Marathahalli - Sarjapur Outer Ring Road
Bangalore, KA  560103, India
Email: csekar@juniper.net

Shahram Davari
Broadcom
3151 Zanker Road
San Jose, CA 95134
Email: davari@broadcom.com

Linda Dunbar
Huawei Technologies
5430 Legacy Drive, Suite #175
Plano, TX 75024
Email: linda.dunbar@huawei.com

Barak Gafni
Mellanox
6 Habarzel St.
Tel Aviv, Israel
Email: gbarak@mellanox.com

Andrew Qu
MediaTek
2860 Junction Ave.
San Jose, CA 95134
Email: andrew.qu@mediatek.com

Jeff Tantsura
Ericsson
200 Holger Way
San Jose, CA 95134
Email: jeff.tantsura@ericsson.com

Yakov Rekhter

Ebben Aries
Facebook
1601 Willow Road
Menlo Park, CA 94025
Email: exa@fb.com

Daniel Voyer
Bell Canada
Email: daniel.voyer@bell.ca

Wen Wang
Century Link
2355 Dulles Corner Blvd.
Herndon, VA 20171
Email: wen.wang@centurylink.com

Himanshu Shah
Ciena
3939 North 1st Street
San Jose, CA 95112
Email: hshah@ciena.com

Ramki Krishnan
Dell


Since it is returning a number coercion shouldn't be a factor. I normally do use ===  in these cases. This method was copied directly from readium's code.


93469234-DemandResponse2 (vidanovic) - Win64-Windows-7-VisualStudio-12: OK (1685 of 1688 tests passed)

![Build Badge](http://img.shields.io/badge/build%20status-passing-green.svg) ![Test Badge](http://img.shields.io/badge/tests%20passed-99.82%25-yellow.svg)


Ack. Feel free to push stuff. I'll wait and sort it out.


It's really not **that** important.

It's just that if there is an error in this test, we'll end up with two times the line `it should execute the command without error` errored instead of only one.
Because `done()` is called twice, at least this is what I suspect.

This is what happen to me when I have an error from the command.
![capture](https://cloud.githubusercontent.com/assets/597828/8445403/a9937fa0-1f6a-11e5-9c48-31e17e216f36.PNG)


If we are modifying the base classes to add the desired functionality (as opposed to subclassing), do we need to still be able to inject custom instances?


@DiamondBond sched_tuner.sh to be exact.

oh well, more micro optimizations to go:p


What does this construct do?


should we rename this?


@hastalafiesta I installed an app called Device Debug, and this is a log obtained from i9070_JB:

http://hastebin.com/ecepotuhof.sm

On Camera 1 you can clearly see that Record stride value for our device is 960, so I don't see the point to set an irregular value like 4096.


Los coding guidelines prohiben usar el estilo de windows :P (todos los braces en la misma línea).


I don't understand the purpose of `commands_mutex_` and `messages_mutex_`. Need a design doc.


Do you mean I should add self.stringObfuscated = True in the constructor?
Wont this give a + only when hex is present in the string?How does it give
false+?

On Wed, Jul 1, 2015 at 5:02 AM, Jose Miguel Esparza <
notifications@github.com> wrote:

> This is ok. But I would remove this code from the PDFHexString object, it
> will give false positives because the hexa strings are already encoded, so
> they will be always there as obfuscated...
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/rohit-dua/peepdf/commit/a396c6343e3460a258756314e452f9e2d45adc68#commitcomment-11943575>
> .
>


Esto no funciona con lenguajes que no sigan el orden al que estamos acostumbrados. Por qué no mejor haces una plantilla en la traducción y reemplazas los campos? Ya lo hacemos en otras partes del código (como `clarificationEmailBody`).


This {name} must be +S((re-evaluated?Yes.))


verifies if*


esto sobra?


FacilityTabGridView (evanweaver) - i386-Windows-7-VisualStudio-12: OK (2191 of 2213 tests passed)

![Build Badge](http://img.shields.io/badge/build%20status-warnings-yellow.svg) ![Test Badge](http://img.shields.io/badge/tests%20passed-99.01%25-yellow.svg)


Esto no se ve muy saludable aquí :P


Nice plug ;)


Why is there no catch statement here?


I don't think a HIL can ever be the PX4FMU_V1 board.


Esto es intencional? Veo que todo lo demás se sigue refiriendo a si es público o no, no a que sea de registro.


this number seems very low... I wonder if there's a scaling issue.


@JenningsPatrick , @freddylukai Here is some of Carson's runtime. But this might answer some of your questions about how we might generate SQL dynamically. Maybe.


Sorry!  Please ignore this pull request:

1. there's a typo in the code
2. define function in if statement is unpredictable

I will make a new pull request.


Here as well, I don't think the PX4FMU_V1 would ever get to this place so the function could just return 500.


@xinqiu 抱歉，你应该去 [Translate 01-introduction first-time-setup v2 by networm · Pull Request #150 · progit/progit2-zh](https://github.com/progit/progit2-zh/pull/150) 审校，这样其他人也可以看到，在这里审校的话只有我能看到。


Really sweet! :+1: Taking it to the next level, and congrats on making the Blackhat USA Arsenal! Great to see a FruityWifi module and excited to see the Wifi Pineapple fitting in too.  So tell me about all the "bunch of evi.. eer fun things to do"... :P Keep it up! :+1:


This is fine, it's a common python idiom: http://stackoverflow.com/questions/431866/ignore-python-multiple-return-value


For the GRIF-16s the event trailer contains the accepted channel id, i.e. the number of requested triggers that made it past several conditions like dead-time, FIFO, etc.
For the 4Gs the event trailer was planned to contain some status information from the NIOS code (which is only output if the errors aren't suppressed). Not sure if this feature will be kept.


Thanks.  I was meaning to ask for this.


this line is only to fix the message if using windows or linux (forgot to do that on the last pull request)


Hi @alluk, thanks for submitting this!

We are going to manage translation work through [Transifex](https://www.transifex.com/projects/p/streetmix/). The goal is to have [a pipeline from Transifex](https://github.com/streetmix/translator) that create these translations. However, this is a great start and I can merge this in.

Would you mind moving `fi.json` to `fi/translation.json` and then updating this pull request? (In the future we will separate strings that part of the Streetmix app and strings that are part of each of the individual segments because that would be easier to manage.)


I created a new grunt task to verify any errors and then fail


Quality division operands right way around here :)


I normally only use the local multicore processing so don't have a ton of experience with the IPython interface directly. It is the same infrastructure we use in bcbio which works cleanly on LSF so I'm not sure why you have so many issues. Remotely debugging other people's cluster is tricky, so I'm not sure if saying it works for me helps much.

Regarding NFS non-support, unless we want to write something where we write temp files locally and then copy over to a shared mount, we'll need to suggest a multicore setup for these types of clusters. This might just be a matter of updating the documentation unless we want to take on a bigger project.


Why use include files with only one line? It is more obvious to me what is going on without using an include here.


You're missing this https://github.com/AICP/packages_apps_Settings/commit/3372920a19b28e0cb81dc6ba197b337028b68d98

There is also a tile commit right after


This should be fine now. Can you please retry?


Thanks for this @MrMatthias. I'll take a look and merge the changes.


Why ordered list ('ol`) and not unordered list (`ul`)?


Another way to do this might be to do away with tar support and gzip the originals in-situ.  I'll look into that tomorrow...


can u give me an example?


This is what I was talking about with sliding boxes etc, images referenced via css...


I don't understand the need for synchronization and an explicit "initialize" method.

1. What's wrong with initializing the SecureRandom as soon as you create it?

2. If initialization needs to be deferred, I think it's better to use a lazy variable and/or lazy method.  http://daily-scala.blogspot.com/2009/09/lazy-val.html


1. See if you can get of the initialize method entirely.

2. If you can't, don't throw an exception.  Just do the initialization.


Double?


.footer{
      position:absolute;
      bottom:30px;left:48%;
}


<div class="footer">©2015<div>


Question for you where is the tmo defconfig?

https://github.com/micky387/android_device_samsung_tbltetmo/blob/master/BoardConfig.mk#L21


nice


hm.. I see. One potential problem is if the stream emits `close` before `error` then the test would pass.
I think this should probably be a bug in mocha because it shouldn't show two logs just because you call the `done` twice.
If you agree, would you open an issue there (so you get the credit)? If not, I'll open one.
I'll leave what it is for now.


Cool.....Have you gotten the refresh to work yet?  I did this with beta 5 or 6 in ObjC.  Also I'm assuming if you do a professional host like heroku the refresh and accessURL's will all be the same?!?