\ Additional tests on the the ANS Forth Core word set

\ This program was written by Gerry Jackson in 2007, with contributions from
\ others where indicated, and is in the public domain - it can be distributed
\ and/or modified in any way but please retain this notice.

\ This program is distributed in the hope that it will be useful,
\ but WITHOUT ANY WARRANTY; without even the implied warranty of
\ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

\ The tests are not claimed to be comprehensive or correct

\ ------------------------------------------------------------------------------
\ Version 0.13 TRUE changed to <TRUE> as defined in core.fr
\         0.12 22 July 2015, >IN Manipulation test modified to work on 16 bit
\              Forth systems
\         0.11 25 April 2015 Number prefixes # $ % and 'c' character input tested
\         0.10 3 August 2014 Test IMMEDIATE doesn't toggle an immediate flag
\         0.3  1 April 2012 Tests placed in the public domain.
\              Testing multiple ELSE's.
\              Further tests on DO +LOOPs.
\              Ackermann function added to test RECURSE.
\              >IN manipulation in interpreter mode
\              Immediate CONSTANTs, VARIABLEs and CREATEd words tests.
\              :NONAME with RECURSE moved to core extension tests.
\              Parsing behaviour of S" ." and ( tested
\         0.2  6 March 2009 { and } replaced with T{ and }T
\              Added extra RECURSE tests
\         0.1  20 April 2007 Created
\ ------------------------------------------------------------------------------
\ The tests are based on John Hayes test program for the core word set
\
\ This file provides some more tests on Core words where the original Hayes
\ tests are thought to be incomplete
\
\ Words tested in this file are:
\     DO +LOOP RECURSE ELSE >IN IMMEDIATE
\ ------------------------------------------------------------------------------
\ Assumptions and dependencies:
\     - tester.fr or ttester.fs has been loaded prior to this file
\     - core.fr has been loaded so that constants <TRUE> MAX-INT, MIN-INT and
\       MAX-UINT are defined
\ ------------------------------------------------------------------------------

DECIMAL

TESTING DO +LOOP with run-time increment, negative increment, infinite loop
\ Contributed by Reinhold Straub

VARIABLE ITERATIONS
VARIABLE INCREMENT
: GD7 ( LIMIT START INCREMENT -- )
   INCREMENT !
   0 ITERATIONS !
   DO
      1 ITERATIONS +!
      I
      ITERATIONS @  6 = IF LEAVE THEN
      INCREMENT @
   +LOOP ITERATIONS @
;

T{  4  4 -1 GD7 -> 4 1 }T
T{  1  4 -1 GD7 -> 4 3 2 1 4 }T
T{  4  1 -1 GD7 -> 1 0 -1 -2 -3 -4 6 }T
T{  4  1  0 GD7 -> 1 1 1 1 1 1 6 }T
T{  0  0  0 GD7 -> 0 0 0 0 0 0 6 }T
T{  1  4  0 GD7 -> 4 4 4 4 4 4 6 }T
T{  1  4  1 GD7 -> 4 5 6 7 8 9 6 }T
T{  4  1  1 GD7 -> 1 2 3 3 }T
T{  4  4  1 GD7 -> 4 5 6 7 8 9 6 }T
T{  2 -1 -1 GD7 -> -1 -2 -3 -4 -5 -6 6 }T
T{ -1  2 -1 GD7 -> 2 1 0 -1 4 }T
T{  2 -1  0 GD7 -> -1 -1 -1 -1 -1 -1 6 }T
T{ -1  2  0 GD7 -> 2 2 2 2 2 2 6 }T
T{ -1  2  1 GD7 -> 2 3 4 5 6 7 6 }T
T{  2 -1  1 GD7 -> -1 0 1 3 }T
T{ -20 30 -10 GD7 -> 30 20 10 0 -10 -20 6 }T
T{ -20 31 -10 GD7 -> 31 21 11 1 -9 -19 6 }T
T{ -20 29 -10 GD7 -> 29 19 9 -1 -11 5 }T

\ ------------------------------------------------------------------------------
TESTING DO +LOOP with large and small increments

\ Contributed by Andrew Haley

MAX-UINT 8 RSHIFT 1+ CONSTANT USTEP
USTEP NEGATE CONSTANT -USTEP
MAX-INT 7 RSHIFT 1+ CONSTANT STEP
STEP NEGATE CONSTANT -STEP

VARIABLE BUMP

T{ : GD8 BUMP ! DO 1+ BUMP @ +LOOP ; -> }T

T{ 0 MAX-UINT 0 USTEP GD8 -> 256 }T
T{ 0 0 MAX-UINT -USTEP GD8 -> 256 }T

T{ 0 MAX-INT MIN-INT STEP GD8 -> 256 }T
T{ 0 MIN-INT MAX-INT -STEP GD8 -> 256 }T

\ Two's complement arithmetic, wraps around modulo wordsize
\ Only tested if the Forth system does wrap around, use of conditional
\ compilation deliberately avoided

MAX-INT 1+ MIN-INT = CONSTANT +WRAP?
MIN-INT 1- MAX-INT = CONSTANT -WRAP?
MAX-UINT 1+ 0=       CONSTANT +UWRAP?
0 1- MAX-UINT =      CONSTANT -UWRAP?

: GD9  ( n limit start step f result -- )
   >R IF GD8 ELSE 2DROP 2DROP R@ THEN -> R> }T
;

T{ 0 0 0  USTEP +UWRAP? 256 GD9
T{ 0 0 0 -USTEP -UWRAP?   1 GD9
T{ 0 MIN-INT MAX-INT  STEP +WRAP? 1 GD9
T{ 0 MAX-INT MIN-INT -STEP -WRAP? 1 GD9

\ ------------------------------------------------------------------------------
TESTING DO +LOOP with maximum and minimum increments

: (-MI) MAX-INT DUP NEGATE + 0= IF MAX-INT NEGATE ELSE -32767 THEN ;
(-MI) CONSTANT -MAX-INT

T{ 0 1 0 MAX-INT GD8  -> 1 }T
T{ 0 -MAX-INT NEGATE -MAX-INT OVER GD8  -> 2 }T

T{ 0 MAX-INT  0 MAX-INT GD8  -> 1 }T
T{ 0 MAX-INT  1 MAX-INT GD8  -> 1 }T
T{ 0 MAX-INT -1 MAX-INT GD8  -> 2 }T
T{ 0 MAX-INT DUP 1- MAX-INT GD8  -> 1 }T

T{ 0 MIN-INT 1+   0 MIN-INT GD8  -> 1 }T
T{ 0 MIN-INT 1+  -1 MIN-INT GD8  -> 1 }T
T{ 0 MIN-INT 1+   1 MIN-INT GD8  -> 2 }T
T{ 0 MIN-INT 1+ DUP MIN-INT GD8  -> 1 }T

\ ------------------------------------------------------------------------------
TESTING multiple RECURSEs in one colon definition

: ACK ( m n -- u )    \ Ackermann function, from Rosetta Code
   OVER 0= IF  NIP 1+ EXIT  THEN       \ ack(0, n) = n+1
   SWAP 1- SWAP                        ( -- m-1 n )
   DUP  0= IF  1+  RECURSE EXIT  THEN  \ ack(m, 0) = ack(m-1, 1)
   1- OVER 1+ SWAP RECURSE RECURSE     \ ack(m, n) = ack(m-1, ack(m,n-1))
;

T{ 0 0 ACK ->  1 }T
T{ 3 0 ACK ->  5 }T
T{ 2 4 ACK -> 11 }T

\ ------------------------------------------------------------------------------
TESTING multiple ELSE's in an IF statement
\ Discussed on comp.lang.forth and accepted as valid ANS Forth

: MELSE IF 1 ELSE 2 ELSE 3 ELSE 4 ELSE 5 THEN ;
T{ 0 MELSE -> 2 4 }T
T{ -1 MELSE -> 1 3 5 }T

\ ------------------------------------------------------------------------------
TESTING manipulation of >IN in interpreter mode

T{ 12345 DEPTH OVER 9 < 34 AND + 3 + >IN ! -> 12345 2345 345 45 5 }T
T{ 14145 8115 ?DUP 0= 34 AND >IN +! TUCK MOD 14 >IN ! GCD CALCULATION -> 15 }T

\ ------------------------------------------------------------------------------
TESTING IMMEDIATE with CONSTANT  VARIABLE and CREATE [ ... DOES> ]

T{ 123 CONSTANT IW1 IMMEDIATE IW1 -> 123 }T
T{ : IW2 IW1 LITERAL ; IW2 -> 123 }T
T{ VARIABLE IW3 IMMEDIATE 234 IW3 ! IW3 @ -> 234 }T
T{ : IW4 IW3 [ @ ] LITERAL ; IW4 -> 234 }T
\ T{ :NONAME [ 345 ] IW3 [ ! ] ; DROP IW3 @ -> 345 }T
T{ CREATE IW5 456 , IMMEDIATE -> }T
\ T{ :NONAME IW5 [ @ IW3 ! ] ; DROP IW3 @ -> 456 }T
T{ : IW6 CREATE , IMMEDIATE DOES> @ 1+ ; -> }T
T{ 111 IW6 IW7 IW7 -> 112 }T
T{ : IW8 IW7 LITERAL 1+ ; IW8 -> 113 }T
T{ : IW9 CREATE , DOES> @ 2 + IMMEDIATE ; -> }T
: FIND-IW BL WORD FIND NIP ;  ( -- 0 | 1 | -1 )
T{ 222 IW9 IW10 FIND-IW IW10 -> -1 }T   \ IW10 is not immediate
T{ IW10 FIND-IW IW10 -> 224 1 }T        \ IW10 becomes immediate

\ ------------------------------------------------------------------------------
TESTING that IMMEDIATE doesn't toggle a flag

VARIABLE IT1 0 IT1 !
: IT2 1234 IT1 ! ; IMMEDIATE IMMEDIATE
T{ : IT3 IT2 ; IT1 @ -> 1234 }T

\ ------------------------------------------------------------------------------
TESTING parsing behaviour of S" ." and (
\ which should parse to just beyond the terminating character no space needed

T{ : GC5 S" A string"2DROP ; GC5 -> }T
T{ ( A comment)1234 -> 1234 }T
T{ : PB1 CR ." You should see 2345: "." 2345"( A comment) CR ; PB1 -> }T

\ ------------------------------------------------------------------------------
TESTING number prefixes # $ % and 'c' character input
\ Adapted from the Forth 200X Draft 14.5 document

VARIABLE OLD-BASE
DECIMAL BASE @ OLD-BASE !
T{ #1289 -> 1289 }T
\ T{ #12346789. -> 12346789. }T
T{ #-1289 -> -1289 }T
\ T{ #-12346789. -> -12346789. }T
T{ $12eF -> 4847 }T
\ T{ $12aBcDeF. -> 313249263. }T
T{ $-12eF -> -4847 }T
\ T{ $-12AbCdEf. -> -313249263. }T
T{ %10010110 -> 150 }T
\ T{ %10010110. -> 150. }T
T{ %-10010110 -> -150 }T
\ T{ %-10010110. -> -150. }T
T{ 'z' -> char z }T
T{ 'Z' -> char Z }T
\ Check BASE is unchanged
T{ BASE @ OLD-BASE @ = -> <TRUE> }T

\ Repeat in Hex mode
16 OLD-BASE ! 16 BASE !
T{ #1289 -> 509 }T
\ { #12346789. -> BC65A5. }T
T{ #-1289 -> -509 }T
\ T{ #-12346789. -> -BC65A5. }T
T{ $12eF -> 12EF }T
\ T{ $12aBcDeF. -> 12AbCdeF. }T
T{ $-12eF -> -12EF }T
\ T{ $-12AbCdEf. -> -12ABCDef. }T
T{ %10010110 -> 96 }T
\ T{ %10010110. -> 96. }T
T{ %-10010110 -> -96 }T
\ T{ %-10010110. -> -96. }T
T{ 'z' -> char z }T
T{ 'Z' -> char Z }T
\ Check BASE is unchanged
T{ BASE @ OLD-BASE @ = -> <TRUE> }T   \ 2

DECIMAL
\ Check number prefixes in compile mode
T{ : nmp  #8327 $-2cbe %011010111 ''' ; nmp -> 8327 -11454 215 39 }T

\ ------------------------------------------------------------------------------

CR .( End of additional Core tests) CR

{'repo_name': 'jkotlinski/durexforth', 'stars': '116', 'repo_language': 'Forth', 'file_name': 'sintab.py', 'mime_type': 'text/plain', 'hash': -7306037004348709206, 'source_dataset': 'data'}


/**
 * Copyright (c) 2015-present, Facebook, Inc.
 *
 * This source code is licensed under the MIT license found in the
 * LICENSE file in the root directory of this source tree.
 */

#import "RCTModuleMethod.h"

#import <objc/message.h>

#import "RCTAssert.h"
#import "RCTBridge+Private.h"
#import "RCTBridge.h"
#import "RCTConvert.h"
#import "RCTCxxConvert.h"
#import "RCTLog.h"
#import "RCTManagedPointer.h"
#import "RCTParserUtils.h"
#import "RCTProfile.h"
#import "RCTUtils.h"

typedef BOOL (^RCTArgumentBlock)(RCTBridge *, NSUInteger, id);

/**
 * Get the converter function for the specified type
 */
static SEL selectorForType(NSString *type)
{
  const char *input = type.UTF8String;
  return NSSelectorFromString([RCTParseType(&input) stringByAppendingString:@":"]);
}

@implementation RCTMethodArgument

- (instancetype)initWithType:(NSString *)type
                 nullability:(RCTNullability)nullability
                      unused:(BOOL)unused
{
  if (self = [super init]) {
    _type = [type copy];
    _nullability = nullability;
    _unused = unused;
  }
  return self;
}

@end

@implementation RCTModuleMethod
{
  Class _moduleClass;
  const RCTMethodInfo *_methodInfo;
  NSString *_JSMethodName;

  SEL _selector;
  NSInvocation *_invocation;
  NSArray<RCTArgumentBlock> *_argumentBlocks;
  NSMutableArray *_retainedObjects;
}

static void RCTLogArgumentError(RCTModuleMethod *method, NSUInteger index,
                                id valueOrType, const char *issue)
{
  RCTLogError(@"Argument %tu (%@) of %@.%s %s", index, valueOrType,
              RCTBridgeModuleNameForClass(method->_moduleClass),
              method.JSMethodName, issue);
}

RCT_NOT_IMPLEMENTED(- (instancetype)init)

RCT_EXTERN_C_BEGIN

// returns YES if the selector ends in a colon (indicating that there is at
// least one argument, and maybe more selector parts) or NO if it doesn't.
static BOOL RCTParseSelectorPart(const char **input, NSMutableString *selector)
{
  NSString *selectorPart;
  if (RCTParseSelectorIdentifier(input, &selectorPart)) {
    [selector appendString:selectorPart];
  }
  RCTSkipWhitespace(input);
  if (RCTReadChar(input, ':')) {
    [selector appendString:@":"];
    RCTSkipWhitespace(input);
    return YES;
  }
  return NO;
}

static BOOL RCTParseUnused(const char **input)
{
  return RCTReadString(input, "__unused") ||
         RCTReadString(input, "__attribute__((unused))");
}

static RCTNullability RCTParseNullability(const char **input)
{
  if (RCTReadString(input, "nullable")) {
    return RCTNullable;
  } else if (RCTReadString(input, "nonnull")) {
    return RCTNonnullable;
  }
  return RCTNullabilityUnspecified;
}

static RCTNullability RCTParseNullabilityPostfix(const char **input)
{
  if (RCTReadString(input, "_Nullable")) {
    return RCTNullable;
  } else if (RCTReadString(input, "_Nonnull")) {
    return RCTNonnullable;
  }
  return RCTNullabilityUnspecified;
}

// returns YES if execution is safe to proceed (enqueue callback invocation), NO if callback has already been invoked
#if RCT_DEBUG
static BOOL checkCallbackMultipleInvocations(BOOL *didInvoke) {
  if (*didInvoke) {
      RCTFatal(RCTErrorWithMessage(@"Illegal callback invocation from native module. This callback type only permits a single invocation from native code."));
      return NO;
  } else {
      *didInvoke = YES;
      return YES;
  }
}
#endif

extern NSString *RCTParseMethodSignature(const char *input, NSArray<RCTMethodArgument *> **arguments);
NSString *RCTParseMethodSignature(const char *input, NSArray<RCTMethodArgument *> **arguments)
{
  RCTSkipWhitespace(&input);

  NSMutableArray *args;
  NSMutableString *selector = [NSMutableString new];
  while (RCTParseSelectorPart(&input, selector)) {
    if (!args) {
      args = [NSMutableArray new];
    }

    // Parse type
    if (RCTReadChar(&input, '(')) {
      RCTSkipWhitespace(&input);

      BOOL unused = RCTParseUnused(&input);
      RCTSkipWhitespace(&input);

      RCTNullability nullability = RCTParseNullability(&input);
      RCTSkipWhitespace(&input);

      NSString *type = RCTParseType(&input);
      RCTSkipWhitespace(&input);
      if (nullability == RCTNullabilityUnspecified) {
        nullability = RCTParseNullabilityPostfix(&input);
      }
      [args addObject:[[RCTMethodArgument alloc] initWithType:type
                                                  nullability:nullability
                                                       unused:unused]];
      RCTSkipWhitespace(&input);
      RCTReadChar(&input, ')');
      RCTSkipWhitespace(&input);
    } else {
      // Type defaults to id if unspecified
      [args addObject:[[RCTMethodArgument alloc] initWithType:@"id"
                                                  nullability:RCTNullable
                                                       unused:NO]];
    }

    // Argument name
    RCTParseArgumentIdentifier(&input, NULL);
    RCTSkipWhitespace(&input);
  }

  *arguments = [args copy];
  return selector;
}

RCT_EXTERN_C_END

- (instancetype)initWithExportedMethod:(const RCTMethodInfo *)exportedMethod
                           moduleClass:(Class)moduleClass
{
  if (self = [super init]) {
    _moduleClass = moduleClass;
    _methodInfo = exportedMethod;
  }
  return self;
}

- (void)processMethodSignature
{
  NSArray<RCTMethodArgument *> *arguments;
  _selector = NSSelectorFromString(RCTParseMethodSignature(_methodInfo->objcName, &arguments));
  RCTAssert(_selector, @"%s is not a valid selector", _methodInfo->objcName);

  // Create method invocation
  NSMethodSignature *methodSignature = [_moduleClass instanceMethodSignatureForSelector:_selector];
  RCTAssert(methodSignature, @"%s is not a recognized Objective-C method.", sel_getName(_selector));
  NSInvocation *invocation = [NSInvocation invocationWithMethodSignature:methodSignature];
  invocation.selector = _selector;
  _invocation = invocation;
  NSMutableArray *retainedObjects = [NSMutableArray array];
  _retainedObjects = retainedObjects;

  // Process arguments
  NSUInteger numberOfArguments = methodSignature.numberOfArguments;
  NSMutableArray<RCTArgumentBlock> *argumentBlocks =
    [[NSMutableArray alloc] initWithCapacity:numberOfArguments - 2];

#if RCT_DEBUG
  __weak RCTModuleMethod *weakSelf = self;
#endif

#define RCT_RETAINED_ARG_BLOCK(_logic) \
[argumentBlocks addObject:^(__unused RCTBridge *bridge, NSUInteger index, id json) { \
  _logic                                                                             \
  [invocation setArgument:&value atIndex:(index) + 2];                               \
  if (value) {                                                                       \
    [retainedObjects addObject:value];                                               \
  }                                                                                \
  return YES;                                                                        \
}]

#define __PRIMITIVE_CASE(_type, _nullable) {                                           \
  isNullableType = _nullable;                                                          \
  _type (*convert)(id, SEL, id) = (__typeof__(convert))objc_msgSend;                   \
  [argumentBlocks addObject:^(__unused RCTBridge *bridge, NSUInteger index, id json) { \
    _type value = convert([RCTConvert class], selector, json);                         \
    [invocation setArgument:&value atIndex:(index) + 2];                               \
    return YES;                                                                        \
  }];                                                                                \
  break;                                                                               \
}

#define PRIMITIVE_CASE(_type) __PRIMITIVE_CASE(_type, NO)
#define NULLABLE_PRIMITIVE_CASE(_type) __PRIMITIVE_CASE(_type, YES)

// Explicitly copy the block
#define __COPY_BLOCK(block...)         \
  id value = [block copy];             \
  if (value) {                         \
    [retainedObjects addObject:value]; \
  }                                    \

#if RCT_DEBUG
#define BLOCK_CASE(_block_args, _block) RCT_RETAINED_ARG_BLOCK(         \
  if (json && ![json isKindOfClass:[NSNumber class]]) {                 \
    RCTLogArgumentError(weakSelf, index, json, "should be a function"); \
    return NO;                                                          \
  }                                                                     \
  __block BOOL didInvoke = NO;                                          \
  __COPY_BLOCK(^_block_args {                                           \
    if (checkCallbackMultipleInvocations(&didInvoke)) _block            \
  });                                                                   \
)
#else
#define BLOCK_CASE(_block_args, _block) \
  RCT_RETAINED_ARG_BLOCK( __COPY_BLOCK(^_block_args { _block }); )
#endif

  for (NSUInteger i = 2; i < numberOfArguments; i++) {
    const char *objcType = [methodSignature getArgumentTypeAtIndex:i];
    BOOL isNullableType = NO;
    RCTMethodArgument *argument = arguments[i - 2];
    NSString *typeName = argument.type;
    SEL selector = selectorForType(typeName);
    if ([RCTConvert respondsToSelector:selector]) {
      switch (objcType[0]) {
        // Primitives
        case _C_CHR: PRIMITIVE_CASE(char)
        case _C_UCHR: PRIMITIVE_CASE(unsigned char)
        case _C_SHT: PRIMITIVE_CASE(short)
        case _C_USHT: PRIMITIVE_CASE(unsigned short)
        case _C_INT: PRIMITIVE_CASE(int)
        case _C_UINT: PRIMITIVE_CASE(unsigned int)
        case _C_LNG: PRIMITIVE_CASE(long)
        case _C_ULNG: PRIMITIVE_CASE(unsigned long)
        case _C_LNG_LNG: PRIMITIVE_CASE(long long)
        case _C_ULNG_LNG: PRIMITIVE_CASE(unsigned long long)
        case _C_FLT: PRIMITIVE_CASE(float)
        case _C_DBL: PRIMITIVE_CASE(double)
        case _C_BOOL: PRIMITIVE_CASE(BOOL)
        case _C_SEL: NULLABLE_PRIMITIVE_CASE(SEL)
        case _C_CHARPTR: NULLABLE_PRIMITIVE_CASE(const char *)
        case _C_PTR: NULLABLE_PRIMITIVE_CASE(void *)

        case _C_ID: {
          isNullableType = YES;
          id (*convert)(id, SEL, id) = (__typeof__(convert))objc_msgSend;
          RCT_RETAINED_ARG_BLOCK(
            id value = convert([RCTConvert class], selector, json);
          );
          break;
        }

        case _C_STRUCT_B: {
          NSMethodSignature *typeSignature = [RCTConvert methodSignatureForSelector:selector];
          NSInvocation *typeInvocation = [NSInvocation invocationWithMethodSignature:typeSignature];
          typeInvocation.selector = selector;
          typeInvocation.target = [RCTConvert class];

          [argumentBlocks addObject:^(__unused RCTBridge *bridge, NSUInteger index, id json) {
            void *returnValue = malloc(typeSignature.methodReturnLength);
            [typeInvocation setArgument:&json atIndex:2];
            [typeInvocation invoke];
            [typeInvocation getReturnValue:returnValue];
            [invocation setArgument:returnValue atIndex:index + 2];
            free(returnValue);
            return YES;
          }];
          break;
        }

        default: {
          static const char *blockType = @encode(__typeof__(^{}));
          if (!strcmp(objcType, blockType)) {
            BLOCK_CASE((NSArray *args), {
              [bridge enqueueCallback:json args:args];
            });
          } else {
            RCTLogError(@"Unsupported argument type '%@' in method %@.",
                        typeName, [self methodName]);
          }
        }
      }
    } else if ([typeName isEqualToString:@"RCTResponseSenderBlock"]) {
      BLOCK_CASE((NSArray *args), {
        [bridge enqueueCallback:json args:args];
      });
    } else if ([typeName isEqualToString:@"RCTResponseErrorBlock"]) {
      BLOCK_CASE((NSError *error), {
        [bridge enqueueCallback:json args:@[RCTJSErrorFromNSError(error)]];
      });
    } else if ([typeName isEqualToString:@"RCTPromiseResolveBlock"]) {
      RCTAssert(i == numberOfArguments - 2,
                @"The RCTPromiseResolveBlock must be the second to last parameter in %@",
                [self methodName]);
      BLOCK_CASE((id result), {
        [bridge enqueueCallback:json args:result ? @[result] : @[]];
      });
    } else if ([typeName isEqualToString:@"RCTPromiseRejectBlock"]) {
      RCTAssert(i == numberOfArguments - 1,
                @"The RCTPromiseRejectBlock must be the last parameter in %@",
                [self methodName]);
      BLOCK_CASE((NSString *code, NSString *message, NSError *error), {
        NSDictionary *errorJSON = RCTJSErrorFromCodeMessageAndNSError(code, message, error);
        [bridge enqueueCallback:json args:@[errorJSON]];
      });
    } else if ([typeName hasPrefix:@"JS::"]) {
      NSString *selectorNameForCxxType =
      [[typeName stringByReplacingOccurrencesOfString:@"::" withString:@"_"]
       stringByAppendingString:@":"];
      selector = NSSelectorFromString(selectorNameForCxxType);

      [argumentBlocks addObject:^(__unused RCTBridge *bridge, NSUInteger index, id json) {
        RCTManagedPointer *(*convert)(id, SEL, id) = (__typeof__(convert))objc_msgSend;
        RCTManagedPointer *box = convert([RCTCxxConvert class], selector, json);

        void *pointer = box.voidPointer;
        [invocation setArgument:&pointer atIndex:index + 2];
        [retainedObjects addObject:box];

        return YES;
      }];
    } else {
      // Unknown argument type
      RCTLogError(@"Unknown argument type '%@' in method %@. Extend RCTConvert to support this type.",
                  typeName, [self methodName]);
    }

#if RCT_DEBUG
    RCTNullability nullability = argument.nullability;
    if (!isNullableType) {
      if (nullability == RCTNullable) {
        RCTLogArgumentError(weakSelf, i - 2, typeName, "is marked as "
                            "nullable, but is not a nullable type.");
      }
      nullability = RCTNonnullable;
    }

    /**
     * Special case - Numbers are not nullable in Android, so we
     * don't support this for now. In future we may allow it.
     */
    if ([typeName isEqualToString:@"NSNumber"]) {
      BOOL unspecified = (nullability == RCTNullabilityUnspecified);
      if (!argument.unused && (nullability == RCTNullable || unspecified)) {
        RCTLogArgumentError(weakSelf, i - 2, typeName,
          [unspecified ? @"has unspecified nullability" : @"is marked as nullable"
           stringByAppendingString: @" but React requires that all NSNumber "
           "arguments are explicitly marked as `nonnull` to ensure "
           "compatibility with Android."].UTF8String);
      }
      nullability = RCTNonnullable;
    }

    if (nullability == RCTNonnullable) {
      RCTArgumentBlock oldBlock = argumentBlocks[i - 2];
      argumentBlocks[i - 2] = ^(RCTBridge *bridge, NSUInteger index, id json) {
        if (json != nil) {
          if (!oldBlock(bridge, index, json)) {
            return NO;
          }
          if (isNullableType) {
            // Check converted value wasn't null either, as method probably
            // won't gracefully handle a nil vallue for a nonull argument
            void *value;
            [invocation getArgument:&value atIndex:index + 2];
            if (value == NULL) {
              return NO;
            }
          }
          return YES;
        }
        RCTLogArgumentError(weakSelf, index, typeName, "must not be null");
        return NO;
      };
    }
#endif
  }

#if RCT_DEBUG
  const char *objcType = _invocation.methodSignature.methodReturnType;
  if (_methodInfo->isSync && objcType[0] != _C_ID) {
    RCTLogError(@"Return type of %@.%s should be (id) as the method is \"sync\"",
                RCTBridgeModuleNameForClass(_moduleClass), self.JSMethodName);
  }
#endif

  _argumentBlocks = argumentBlocks;
}

- (SEL)selector
{
  if (_selector == NULL) {
    RCT_PROFILE_BEGIN_EVENT(RCTProfileTagAlways, @"", (@{ @"module": NSStringFromClass(_moduleClass),
                                                          @"method": @(_methodInfo->objcName) }));
    [self processMethodSignature];
    RCT_PROFILE_END_EVENT(RCTProfileTagAlways, @"");
  }
  return _selector;
}

- (const char *)JSMethodName
{
  NSString *methodName = _JSMethodName;
  if (!methodName) {
    const char *jsName = _methodInfo->jsName;
    if (jsName && strlen(jsName) > 0) {
      methodName = @(jsName);
    } else {
      methodName = @(_methodInfo->objcName);
      NSRange colonRange = [methodName rangeOfString:@":"];
      if (colonRange.location != NSNotFound) {
        methodName = [methodName substringToIndex:colonRange.location];
      }
      methodName = [methodName stringByTrimmingCharactersInSet:[NSCharacterSet whitespaceAndNewlineCharacterSet]];
      RCTAssert(methodName.length, @"%s is not a valid JS function name, please"
                " supply an alternative using RCT_REMAP_METHOD()", _methodInfo->objcName);
    }
    _JSMethodName = methodName;
  }
  return methodName.UTF8String;
}

- (RCTFunctionType)functionType
{
  if (strstr(_methodInfo->objcName, "RCTPromise") != NULL) {
    RCTAssert(!_methodInfo->isSync, @"Promises cannot be used in sync functions");
    return RCTFunctionTypePromise;
  } else if (_methodInfo->isSync) {
    return RCTFunctionTypeSync;
  } else {
    return RCTFunctionTypeNormal;
  }
}

- (id)invokeWithBridge:(RCTBridge *)bridge
                module:(id)module
             arguments:(NSArray *)arguments
{
  if (_argumentBlocks == nil) {
    [self processMethodSignature];
  }

#if RCT_DEBUG
  // Sanity check
  RCTAssert([module class] == _moduleClass, @"Attempted to invoke method \
            %@ on a module of class %@", [self methodName], [module class]);

  // Safety check
  if (arguments.count != _argumentBlocks.count) {
    NSInteger actualCount = arguments.count;
    NSInteger expectedCount = _argumentBlocks.count;

    // Subtract the implicit Promise resolver and rejecter functions for implementations of async functions
    if (self.functionType == RCTFunctionTypePromise) {
      actualCount -= 2;
      expectedCount -= 2;
    }

    RCTLogError(@"%@.%s was called with %lld arguments but expects %lld arguments. "
                @"If you haven\'t changed this method yourself, this usually means that "
                @"your versions of the native code and JavaScript code are out of sync. "
                @"Updating both should make this error go away.",
                RCTBridgeModuleNameForClass(_moduleClass), self.JSMethodName,
                (long long)actualCount, (long long)expectedCount);
    return nil;
  }
#endif

  // Set arguments
  NSUInteger index = 0;
  for (id json in arguments) {
    RCTArgumentBlock block = _argumentBlocks[index];
    if (!block(bridge, index, RCTNilIfNull(json))) {
      // Invalid argument, abort
      RCTLogArgumentError(self, index, json, "could not be processed. Aborting method call.");
      return nil;
    }
    index++;
  }

  // Invoke method
#ifdef RCT_MAIN_THREAD_WATCH_DOG_THRESHOLD
  if (RCTIsMainQueue()) {
    CFTimeInterval start = CACurrentMediaTime();
    [_invocation invokeWithTarget:module];
    CFTimeInterval duration = CACurrentMediaTime() - start;
    if (duration > RCT_MAIN_THREAD_WATCH_DOG_THRESHOLD) {
      RCTLogWarn(
                 @"Main Thread Watchdog: Invocation of %@ blocked the main thread for %dms. "
                 "Consider using background-threaded modules and asynchronous calls "
                 "to spend less time on the main thread and keep the app's UI responsive.",
                 [self methodName],
                 (int)(duration * 1000)
                 );
    }
  } else {
    [_invocation invokeWithTarget:module];
  }
#else
  [_invocation invokeWithTarget:module];
#endif

  index = 2;
  [_retainedObjects removeAllObjects];

  if (_methodInfo->isSync) {
    void *returnValue;
    [_invocation getReturnValue:&returnValue];
    return (__bridge id)returnValue;
  }
  return nil;
}

- (NSString *)methodName
{
  if (!_selector) {
    [self processMethodSignature];
  }
  return [NSString stringWithFormat:@"-[%@ %s]", _moduleClass, sel_getName(_selector)];
}

- (NSString *)description
{
  return [NSString stringWithFormat:@"<%@: %p; exports %@ as %s(); type: %s>",
          [self class], self, [self methodName], self.JSMethodName, RCTFunctionDescriptorFromType(self.functionType)];
}

@end

{'repo_name': 'notjosh/Marzipants', 'stars': '144', 'repo_language': 'Objective-C', 'file_name': 'BUCK', 'mime_type': 'text/plain', 'hash': 1435105947304627854, 'source_dataset': 'data'}


|NumFeatures 28 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29 0 0 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 2 4 2 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 53 0 0 |sex 0 1 |pclass 1 0 0 |survived 0 1
|NumFeatures 1 1 1 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 1 1 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 17 0 0 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 17 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 39 0 2 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 30 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 22 0 0 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 24 2 3 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 51 0 1 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 8 3 1 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 16 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 31 1 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 42 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 20 1 0 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 32 0 0 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 28 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 21 1 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 30 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 0.9167 1 2 |sex 0 1 |pclass 1 0 0 |survived 0 1
|NumFeatures 17 1 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 23 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 21 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 14 4 1 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 25 1 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 18 0 0 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 52 1 1 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 19 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 36 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 47 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 25 0 0 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 29.8811345124283 0 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 26 1 1 |sex 1 0 |pclass 0 1 0 |survived 1 0
|NumFeatures 34 0 0 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 21 0 0 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 21 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 64 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 18 1 1 |sex 1 0 |pclass 0 1 0 |survived 1 0
|NumFeatures 28 3 2 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 30 0 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 32 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 22 1 0 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 31 1 1 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 48 0 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 36 1 2 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 70.5 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 0.75 2 1 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 36 1 2 |sex 0 1 |pclass 1 0 0 |survived 0 1
|NumFeatures 29.8811345124283 1 2 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 16 0 0 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 37 1 1 |sex 0 1 |pclass 1 0 0 |survived 0 1
|NumFeatures 29.8811345124283 0 2 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 50 1 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 40 1 1 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 29.8811345124283 0 0 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 19 0 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 29.8811345124283 1 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 21 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 22 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 43 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 27 1 0 |sex 1 0 |pclass 0 1 0 |survived 1 0
|NumFeatures 54 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 20 0 0 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 55 0 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 15 1 1 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 26 0 0 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 55.5 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 21 2 2 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 45 0 0 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 25 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 74 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 20 1 1 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 18 0 0 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 18 0 0 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 32 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 59 2 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 36 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 26 0 0 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 45 1 1 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 4 0 1 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 52 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 18 1 1 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 24.5 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 24 0 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 63 1 0 |sex 1 0 |pclass 1 0 0 |survived 1 0
|NumFeatures 22 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 71 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 24 3 2 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 10 0 2 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 28 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 6 0 2 |sex 0 1 |pclass 1 0 0 |survived 0 1
|NumFeatures 29.8811345124283 0 0 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 39 0 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 45 1 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 26.5 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 40 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 9 0 1 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 41 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 22 0 1 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 4 0 2 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 40 1 1 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 1 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 44 0 0 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 29.8811345124283 1 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 24 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 45 1 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 25 1 2 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 54 1 1 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 35 1 1 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 58 0 2 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 41 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 7 1 1 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 19 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 37 1 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 44 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 5 4 2 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 36 1 0 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 23 0 0 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 60 1 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 1 0 0 |survived 0 1
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 32 0 0 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 35 0 0 |sex 0 1 |pclass 1 0 0 |survived 0 1
|NumFeatures 45 1 1 |sex 0 1 |pclass 1 0 0 |survived 0 1
|NumFeatures 48 1 1 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 19 0 0 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 39 1 5 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 47 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 38 4 2 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 13 2 2 |sex 0 1 |pclass 1 0 0 |survived 0 1
|NumFeatures 21 0 1 |sex 0 1 |pclass 1 0 0 |survived 0 1
|NumFeatures 16 0 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 0.3333 0 2 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 36 0 0 |sex 1 0 |pclass 1 0 0 |survived 1 0
|NumFeatures 40 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 29 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 26 0 0 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 26 2 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 34.5 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 20 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 18 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 2 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 33 1 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 36 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 36 0 0 |sex 0 1 |pclass 1 0 0 |survived 0 1
|NumFeatures 20 0 0 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 48 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 18 0 0 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 42 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 24 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 25 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 20 1 0 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 29.8811345124283 0 0 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 17 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 22 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 31 0 0 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 52 0 0 |sex 0 1 |pclass 1 0 0 |survived 0 1
|NumFeatures 34 0 0 |sex 0 1 |pclass 0 1 0 |survived 0 1
|NumFeatures 30 0 0 |sex 0 1 |pclass 0 1 0 |survived 0 1
|NumFeatures 25 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 38 0 0 |sex 1 0 |pclass 0 1 0 |survived 1 0
|NumFeatures 18 1 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 17 4 2 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 53 1 1 |sex 0 1 |pclass 1 0 0 |survived 0 1
|NumFeatures 22 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 48 0 2 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 61 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 44 1 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 44 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 2 4 1 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 23 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 28 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 60.5 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 1 0 |pclass 0 0 1 |survived 0 1
|NumFeatures 60 0 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 21 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 38.5 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 14 0 0 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 22 0 1 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 50 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 26 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 2 3 1 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 35 1 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 20 0 0 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 19 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 8 1 1 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 50 1 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 19 1 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 33 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 33 0 0 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 16 2 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 1 9 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 16 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 35 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 24 0 0 |sex 1 0 |pclass 0 1 0 |survived 1 0
|NumFeatures 17 2 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 25 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 25 0 0 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 45 1 4 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 28 0 0 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 11 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 39 1 1 |sex 1 0 |pclass 1 0 0 |survived 0 1
|NumFeatures 23 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 24 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 42 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 28 1 0 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 27 0 0 |sex 0 1 |pclass 0 0 1 |survived 0 1
|NumFeatures 29.8811345124283 1 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 14 5 2 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29 1 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 1 0 0 |survived 0 1
|NumFeatures 29 1 0 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 28 0 0 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 32.5 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 22 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 50 0 1 |sex 1 0 |pclass 0 1 0 |survived 0 1
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 31 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 1 0 |pclass 0 0 1 |survived 1 0
|NumFeatures 29 1 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 29.8811345124283 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 31 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 22 2 0 |sex 0 1 |pclass 0 1 0 |survived 1 0
|NumFeatures 28 0 0 |sex 0 1 |pclass 0 0 1 |survived 1 0
|NumFeatures 48 0 0 |sex 0 1 |pclass 1 0 0 |survived 1 0
|NumFeatures 23 0 0 |sex 0 1 |pclass 0 1 0 |survived 1 0

{'repo_name': 'bhrnjica/anndotnet', 'stars': '106', 'repo_language': 'Rich Text Format', 'file_name': 'iris_with_hot_vector_test.txt', 'mime_type': 'text/plain', 'hash': 669371038174536944, 'source_dataset': 'data'}


name: doctoolchain
title: docToolchain
version: 1.0.0
start_page: manual.adoc
#nav:
#- modules/ROOT/nav.adoc

{'repo_name': 'docToolchain/docToolchain', 'stars': '289', 'repo_language': 'Groovy', 'file_name': 'custom-theme.yml', 'mime_type': 'text/plain', 'hash': -8477773792776687395, 'source_dataset': 'data'}


/**
 * Copyright 2015 @ to2.net.
 * name : exchange
 * author : jarryliu
 * date : 2016-07-16 14:52
 * description :
 * history :
 */
package afterSales

type (
	// 换货单接口
	IExchangeOrder interface {
		// 将换货的商品重新发货
		ExchangeShip(spName string, spOrder string) error

		// 消费者延长收货时间
		LongReceive() error

		// 接收换货
		ExchangeReceive() error
	}

	// 换货单
	ExchangeOrder struct {
		// 编号
		Id int32 `db:"id" pk:"yes" auto:"no"`
		// 是否发货
		IsShipped int `db:"is_shipped"`
		// 快递名称
		ShipSpName string `db:"sp_name"`
		// 快递编号
		ShipSpOrder string `db:"sp_order"`
		// 发货时间
		ShipTime int64 `db:"ship_time"`
		// 是否收货
		IsReceived int `db:"is_received"`
		// 收货时间
		ReceiveTime int64 `db:"receive_time"`
	}
)

{'repo_name': 'ixre/go2o', 'stars': '633', 'repo_language': 'Go', 'file_name': 'go2o-publish.sh', 'mime_type': 'text/x-shellscript', 'hash': 25915641650748064, 'source_dataset': 'data'}


/*******************************************************************************
 * This file is part of OpenNMS(R).
 *
 * Copyright (C) 2013-2016 The OpenNMS Group, Inc.
 * OpenNMS(R) is Copyright (C) 1999-2016 The OpenNMS Group, Inc.
 *
 * OpenNMS(R) is a registered trademark of The OpenNMS Group, Inc.
 *
 * OpenNMS(R) is free software: you can redistribute it and/or modify
 * it under the terms of the GNU Affero General Public License as published
 * by the Free Software Foundation, either version 3 of the License,
 * or (at your option) any later version.
 *
 * OpenNMS(R) is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU Affero General Public License for more details.
 *
 * You should have received a copy of the GNU Affero General Public License
 * along with OpenNMS(R).  If not, see:
 *      http://www.gnu.org/licenses/
 *
 * For more information contact:
 *     OpenNMS(R) Licensing <license@opennms.org>
 *     http://www.opennms.org/
 *     http://www.opennms.com/
 *******************************************************************************/

package org.opennms.netmgt.config.wmi;

import java.text.ParseException;
import java.util.Arrays;
import java.util.Collection;

import org.junit.runners.Parameterized.Parameters;
import org.opennms.core.test.xml.XmlTestNoCastor;
import org.opennms.netmgt.collection.api.AttributeType;

public class WmiDatacollectionConfigTest extends XmlTestNoCastor<WmiDatacollectionConfig> {

    public WmiDatacollectionConfigTest(final WmiDatacollectionConfig sampleObject,
            final String sampleXml, final String schemaFile) {
        super(sampleObject, sampleXml, schemaFile);
    }

    @Parameters
    public static Collection<Object[]> data() throws ParseException {
        WmiDatacollectionConfig config = new WmiDatacollectionConfig();
        config.setRrdRepository("/opt/opennms/rrd/snmp/");

        Rrd rrd = new Rrd();
        rrd.setStep(300);
        rrd.getRra().add("RRA:AVERAGE:0.5:1:2016");

        WmiCollection wmiCollection = new WmiCollection();
        wmiCollection.setName("not-default");
        wmiCollection.setRrd(rrd);

        Wpm wpm = new Wpm();
        wpm.setName("wmiOSMemory");
        wpm.setWmiClass("Win32_PerfFormattedData_PerfOS_Memory");
        wpm.setWmiNamespace("root/cimv2");
        wpm.setKeyvalue("Name");
        wpm.setRecheckInterval(3600000);
        wpm.setIfType("all");
        wpm.setResourceType("node");
        wmiCollection.addWpm(wpm);

        Attrib attrib = new Attrib();
        attrib.setName("AvailableBytes");
        attrib.setAlias("wmiOSMemAvailBytes");
        attrib.setWmiObject("AvailableBytes");
        attrib.setType(AttributeType.GAUGE);
        wpm.getAttribs().add(attrib);

        config.getWmiCollections().add(wmiCollection);

        return Arrays.asList(new Object[][] { {
            config,
            "<wmi-datacollection-config rrdRepository=\"/opt/opennms/rrd/snmp/\">" +
                 "<wmi-collection name=\"not-default\">" +
                     "<rrd step=\"300\">" +
                         "<rra>RRA:AVERAGE:0.5:1:2016</rra>" +
                     "</rrd>" +
                     "<wpms>" +
                         "<wpm name=\"wmiOSMemory\" wmiClass=\"Win32_PerfFormattedData_PerfOS_Memory\" wmiNamespace=\"root/cimv2\" " +
                             " keyvalue=\"Name\" recheckInterval=\"3600000\" ifType=\"all\" resourceType=\"node\">" +
                             "<attrib name=\"AvailableBytes\" alias=\"wmiOSMemAvailBytes\" wmiObject=\"AvailableBytes\" type=\"GAUGE\"/>" +
                         "</wpm>" +
                     "</wpms>" +
                 "</wmi-collection>" +
            "</wmi-datacollection-config>",
            "target/classes/xsds/wmi-datacollection.xsd", }, });
    }
}

{'repo_name': 'OpenNMS/opennms', 'stars': '506', 'repo_language': 'Java', 'file_name': 'ReportParameterBuilder.java', 'mime_type': 'text/x-java', 'hash': -650640352615131300, 'source_dataset': 'data'}


package com.salesforce.dva.argus.service.alert.notifier;

import static org.junit.Assert.assertEquals;
import static org.mockito.Mockito.*;

import java.text.MessageFormat;
import java.util.ArrayList;
import java.util.stream.IntStream;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;

import javax.persistence.EntityManager;

import com.salesforce.dva.argus.TestUtils;

import com.salesforce.dva.argus.service.RefocusService;
import com.salesforce.dva.argus.service.HistoryService;
import com.salesforce.dva.argus.entity.History;
import com.salesforce.dva.argus.system.SystemConfiguration;
import com.salesforce.dva.argus.system.SystemMain;
import org.junit.*;
import org.junit.runner.RunWith;
import org.mockito.Mock;

import org.apache.http.impl.client.*;
import org.apache.http.HttpStatus;

import java.util.function.Supplier;
import java.util.Iterator;

import com.google.gson.JsonObject;
import com.google.gson.JsonParser;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.inject.Provider;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;
import java.io.InterruptedIOException;

@RunWith(org.mockito.junit.MockitoJUnitRunner.class)
public class RefocusForwarderTest {

	@Mock private Provider<EntityManager> _emProviderMock;
	@Mock private HistoryService _historyServiceMock;
	@Mock private ObjectMapper _mapper;

    private SystemMain system;
    private SystemConfiguration _configuration;
	private RefocusForwarder refocusForwarder;

    private final Logger LOGGER = LoggerFactory.getLogger(RefocusForwarderTest.class);
    private static final boolean mapToErrorLogging = false;  // call LOGGER.error for all logs when true.

	// @Before
	public void refocus_setup() {

		_configuration = system.getConfiguration();

        _configuration.setProperty("system.property.refocus.enabled", "true");
        _configuration.setProperty(RefocusProperty.REFOCUS_MAX_REQUESTS_PER_MINUTE.getName(), "100000"); // any value above 6000 will result in no sleeping in the unit tests.
        _configuration.setProperty(RefocusProperty.REFOCUS_SEND_INTERVAL_MS.getName(), "0");             // no waiting
        _configuration.setProperty(RefocusProperty.REFOCUS_MAX_BULK_ITEMS.getName(), "50");              // 50 samples per bulk request
        _configuration.setProperty(RefocusProperty.REFOCUS_CONNECTION_REFRESH_MAX_TIMES.getName(), "3"); // ensure it is at least 3

        refocusForwarder = new RefocusForwarder(_configuration, _emProviderMock);

        CloseableHttpClient mockClient = mock(CloseableHttpClient.class);
        refocusForwarder = spy(refocusForwarder);
        when(refocusForwarder.getHttpClient()).thenReturn(mockClient);
	}

    @BeforeClass
    static public void setUpClass() {
    }

    @AfterClass
    static public void tearDownClass() {
    }

    @Before
    public void setup() {

        system = TestUtils.getInstanceWithInMemProps(); // getInstance();
        system.start();  // 4s initially, 0.7s each subsequent run.
        refocus_setup(); // 50ms initially, 0-2ms each subsequent run.
    }

    @After
    public void tearDown() {
        if (system != null) {
            system.getServiceFactory().getManagementService().cleanupRecords();
            system.stop();
        }
    }

	// -----------------------------------------------------------------------------------
    // Support for Generating sequences of stubbed responses.
    // -----------------------------------------------------------------------------------

	static class StubResult {
	    int sleep_ms;
	    int  resp_code;
	    JsonObject gson;

	    public StubResult(int sleep_ms, int resp_code, JsonObject gson)
        {
            this.sleep_ms = sleep_ms;
            this.resp_code = resp_code;
            this.gson = gson;
        }

        public void action() throws RuntimeException
        {
        }
    }

    static class StubIOExceptionResult extends StubResult {

	    public StubIOExceptionResult(int sleep_ms, int resp_code, JsonObject gson)
        {
            super(sleep_ms, resp_code, gson);
        }

        @Override
        public void action() throws RuntimeException
        {
            throw new RuntimeException(new IOException());
        }
    }

    static class StubInterruptedIOExceptionResult extends StubResult {

        public StubInterruptedIOExceptionResult(int sleep_ms, int resp_code, JsonObject gson)
        {
            super(sleep_ms, resp_code, gson);
        }

        @Override
        public void action() throws RuntimeException
        {
            throw new RuntimeException(new InterruptedIOException());
        }
    }

    static JsonObject makeUpcertResponse(String status, int jobId) {
        JsonObject response = new JsonObject();
        response.addProperty("status", status);
        response.addProperty("jobId", jobId);
        return response;
    }

    static StubResult make200(int sleep_ms, int txId)
    {
        return new StubResult(sleep_ms, HttpStatus.SC_OK, makeUpcertResponse("OK", txId));
    }

    // OK result
    static StubResult make201(int sleep_ms, int txId)
    {
        return new StubResult(sleep_ms, HttpStatus.SC_CREATED, makeUpcertResponse("OK", txId));
    }

    static StubResult make204(int sleep_ms, int txId)
    {
        return new StubResult(sleep_ms, HttpStatus.SC_NO_CONTENT, makeUpcertResponse("OK", txId));
    }

    static StubResult makeTimedOut(int sleep_ms)  // suggest timeouts >= 10000
    {
        return new StubResult(sleep_ms, HttpStatus.SC_REQUEST_TIMEOUT, new JsonObject());
    }

    static StubResult makeTooManyRequests(int sleep_ms)
    {
        return new StubResult(sleep_ms, 429, new JsonObject());
    }

    static StubResult makeWithResponseCode(int sleep_ms, int respCode)
    {
        return new StubResult(sleep_ms, respCode, new JsonObject());
    }

    static StubResult makeWithIOException(int sleep_ms, int respCode)
    {
        return new StubIOExceptionResult(sleep_ms, respCode, new JsonObject());
    }

    static StubResult makeWithInterruptedIOException(int sleep_ms, int respCode)
    {
        return new StubInterruptedIOExceptionResult(sleep_ms, respCode, new JsonObject());
    }

    static class ResultStubSupplier implements Supplier<RefocusForwarder.RefocusResponse> {

        protected JsonObject cloneJson(JsonObject r)
        {
            String jsonText = r.toString();
            JsonObject obj = (new JsonParser()).parse(jsonText).getAsJsonObject();
            return obj;
        }

        public RefocusForwarder.RefocusResponse get() throws RuntimeException
        {
            return null;
        }
    }

	static class ResultListSupplier extends ResultStubSupplier {

	    ArrayList<StubResult> results;
	    Iterator<StubResult> iterator;
	    RefocusForwarder forwarder;
	    StubResult defaultResult;

	    public ResultListSupplier( ArrayList<StubResult> resultsToReturn, StubResult defaultResult, RefocusForwarder forwarder)
        {
            super();
            this.results = resultsToReturn;
            this.iterator = resultsToReturn.iterator();
            this.forwarder = forwarder;
            this.defaultResult = defaultResult;
        }

        @Override
	    public RefocusForwarder.RefocusResponse get() throws RuntimeException
        {
            StubResult r = this.iterator.hasNext() ? this.iterator.next() : defaultResult;
            // SleepMs(r.sleep_ms);
            r.action();
            RefocusForwarder.RefocusResponse refocus_response = forwarder.makeResponse(r.resp_code, cloneJson(r.gson));
            return refocus_response;
        }
    }

    static class ResultCycleStubSupplier extends ResultStubSupplier {

        StubResult[] resultCycle;
        Iterator<StubResult> iterator;
        RefocusForwarder forwarder;
        int pos = 0;

        public ResultCycleStubSupplier( StubResult[] resultCycle, RefocusForwarder forwarder)
        {
            this.resultCycle = resultCycle;
            this.pos = 0;
            this.forwarder = forwarder;
            assert(this.resultCycle.length > 0);
        }

        @Override
        public RefocusForwarder.RefocusResponse get() throws RuntimeException
        {
            StubResult r = this.resultCycle[ this.pos % this.resultCycle.length ];
            this.pos = (this.pos + 1) % this.resultCycle.length;
            // SleepMs(r.sleep_ms);
            r.action();
            RefocusForwarder.RefocusResponse refocus_response = forwarder.makeResponse(r.resp_code, cloneJson(r.gson));
            return refocus_response;
        }
    }

    void enqueueSamples(int numSamples, History history)
    {
        IntStream.range(0,numSamples).forEach( x -> {

            String value = String.format("%d", x);
            String sa = String.format("a.b%d|c", x);
            try
            {
                refocusForwarder.sendRefocusNotification(sa, value, "myuser", "mytoken", history);
            }
            catch (Exception e) {
                throw new RuntimeException(e);
            }
        });
    }

    boolean waitUntilQueuedNotificationsAreProcessed(RefocusService service)
    {
        boolean rv = true;
        try
        {
            while( refocusForwarder.getNotificationsDelivered() + refocusForwarder.getNotificationsDiscarded() < refocusForwarder.getNotificationsEnqueued())
            {
                int forwarded = service.forwardNotifications();
            }
        }
        catch( InterruptedException | RuntimeException e)
        {
            rv = false;
        }
        return false;
    }

    void logForwarderStats()
    {
        int maxQueueLength = refocusForwarder.getMaxQueueLength();
        _info(MessageFormat.format("MaxQueueLength was {0} samples", maxQueueLength));
    }

    void verifyProcessed( long expect_delivered, long expect_discarded, long enqueued)
    {
        long delivered = refocusForwarder.getNotificationsDelivered();
        long discarded = refocusForwarder.getNotificationsDiscarded();
        long processed = refocusForwarder.getNotificationsProcessed();
        assertEquals(expect_delivered, delivered);
        assertEquals(expect_discarded, discarded);
        assertEquals(processed, delivered+discarded);
        assertEquals(processed, enqueued);
    }

	// -----------------------------------------------------------------------------------
	// Tests
	// -----------------------------------------------------------------------------------

	@Test
	public void testRefocusForwarderSendSamples() {

        ResultListSupplier responseSupplier = new ResultListSupplier(new ArrayList<StubResult>(),
                make200(10, 1),
                refocusForwarder);

        refocusForwarder.setStubSender(responseSupplier);
        int num_notifications = 100;

		History history = mock(History.class);
        enqueueSamples(num_notifications, history);
        waitUntilQueuedNotificationsAreProcessed(refocusForwarder);

        logForwarderStats();
        verifyProcessed(num_notifications, 0, num_notifications);
	}

    @Test
    public void testRefocusForwarderThrottled() {

        ResultListSupplier responseSupplier = new ResultListSupplier(
                new ArrayList<StubResult>(),
                makeTooManyRequests(120),
                refocusForwarder);

        refocusForwarder.setStubSender(responseSupplier);
        int num_notifications = 5;

        History history = mock(History.class);
        enqueueSamples(num_notifications, history);
        waitUntilQueuedNotificationsAreProcessed(refocusForwarder);

        logForwarderStats();
        verifyProcessed(0, num_notifications, num_notifications);
    }

    @Test
    public void testRefocusForwarderTimedOut() {

        ResultListSupplier responseSupplier = new ResultListSupplier(
                new ArrayList<StubResult>(),
                makeTimedOut(10),
                refocusForwarder);

        refocusForwarder.setStubSender(responseSupplier);
        int num_notifications = 5;

        History history = mock(History.class);
        enqueueSamples(num_notifications, history);
        waitUntilQueuedNotificationsAreProcessed(refocusForwarder);

        logForwarderStats();
        verifyProcessed(0, num_notifications, num_notifications);
    }

    @Test
    public void testRefocusForwarderInterruptedIOException() {

        RefocusForwarder.Duration d = new RefocusForwarder.Duration();

        ResultListSupplier responseSupplier = new ResultListSupplier(
                new ArrayList<StubResult>(),
                makeWithInterruptedIOException(10, -1),
                refocusForwarder);

        refocusForwarder.setStubSender(responseSupplier);
        int num_notifications = 5;

        History history = mock(History.class);
        enqueueSamples(num_notifications, history);
        waitUntilQueuedNotificationsAreProcessed(refocusForwarder);

        logForwarderStats();
        verifyProcessed(0, num_notifications, num_notifications);
    }

    @Test
    public void testRefocusForwarderIOException() {

        ResultListSupplier responseSupplier = new ResultListSupplier(
                new ArrayList<StubResult>(),
                makeWithIOException(10, -1),
                refocusForwarder);

        refocusForwarder.setStubSender(responseSupplier);

        int num_notifications = 5;

        History history = mock(History.class);
        enqueueSamples(num_notifications, history);
        waitUntilQueuedNotificationsAreProcessed(refocusForwarder);

        logForwarderStats();
        verifyProcessed(0, num_notifications, num_notifications);
    }

    @Test
    public void testRefocusNoAuth() {

        ResultListSupplier responseSupplier = new ResultListSupplier(
                    new ArrayList<StubResult>(),
                    makeWithResponseCode(10, HttpStatus.SC_UNAUTHORIZED),
                    refocusForwarder);

        refocusForwarder.setStubSender(responseSupplier);

        int num_notifications = 5;
        History history = mock(History.class);
        enqueueSamples(num_notifications, history);
        waitUntilQueuedNotificationsAreProcessed(refocusForwarder);

        logForwarderStats();
        verifyProcessed(0, num_notifications, num_notifications);
    }

    @Test
    public void testRefocusServiceUnavail() {

        ResultListSupplier responseSupplier = new ResultListSupplier(
                new ArrayList<StubResult>(),
                makeWithResponseCode(10, HttpStatus.SC_SERVICE_UNAVAILABLE),
                refocusForwarder);

        refocusForwarder.setStubSender(responseSupplier);

        int num_notifications = 5;

        History history = mock(History.class);
        enqueueSamples(num_notifications, history);
        waitUntilQueuedNotificationsAreProcessed(refocusForwarder);

        logForwarderStats();
        verifyProcessed(0, num_notifications, num_notifications);
    }

    @Test
    public void testIntermittentThrottle() {

        // Assumes retries >= 3
        StubResult [] cycle = {
                makeTooManyRequests(10),
                makeTooManyRequests(10),
                make200(10, 1)
        };
        ResultCycleStubSupplier responseSupplier = new ResultCycleStubSupplier( cycle, refocusForwarder);
        refocusForwarder.setStubSender(responseSupplier);

        int num_notifications = 5;

        History history = mock(History.class);
        enqueueSamples(num_notifications, history);
        waitUntilQueuedNotificationsAreProcessed(refocusForwarder);

        logForwarderStats();
        verifyProcessed(num_notifications, 0, num_notifications);
    }

    @Test
    public void testIntermittentInterruptedIOException() {

        // Assumes retries >= 3
        StubResult [] cycle = {
                makeWithInterruptedIOException(10, -1),
                makeWithInterruptedIOException(10, -1),
                make200(10, 1)
        };
        ResultCycleStubSupplier responseSupplier = new ResultCycleStubSupplier( cycle, refocusForwarder);
        refocusForwarder.setStubSender(responseSupplier);

        int num_notifications = 5;

        History history = mock(History.class);
        enqueueSamples(num_notifications, history);
        waitUntilQueuedNotificationsAreProcessed(refocusForwarder);

        logForwarderStats();
        verifyProcessed(num_notifications, 0, num_notifications);
    }

	// =================================================================================
    // Logging wrappers
    // Note: these exist because I haven't found a mechanism for setting the log level programmatically.
    // =================================================================================

    // Syntactic sugar
    private void _info(String msg)
    {
        if (mapToErrorLogging) {
            LOGGER.error(msg);
        }
        else {
            LOGGER.info(msg);
        }
    }

    private void _error(String msg)
    {
        LOGGER.error(msg);
    }

    private void _warn(String msg)
    {
        if (mapToErrorLogging) {
            LOGGER.error(msg);
        }
        else {
            LOGGER.warn(msg);
        }
    }

}

{'repo_name': 'salesforce/Argus', 'stars': '453', 'repo_language': 'Java', 'file_name': 'TSDBEntityTest.java', 'mime_type': 'text/x-java', 'hash': -2182217159825362301, 'source_dataset': 'data'}


/* cspt01.f -- translated by f2c (version 20061008).
   You must link the resulting object file with libf2c:
	on Microsoft Windows system, link with libf2c.lib;
	on Linux or Unix systems, link with .../path/to/libf2c.a -lm
	or, if you install libf2c.a in a standard place, with -lf2c -lm
	-- in that order, at the end of the command line, as in
		cc *.o -lf2c -lm
	Source for libf2c is in /netlib/f2c/libf2c.zip, e.g.,

		http://www.netlib.org/f2c/libf2c.zip
*/

#include "f2c.h"
#include "blaswrap.h"

/* Table of constant values */

static complex c_b1 = {0.f,0.f};
static complex c_b2 = {1.f,0.f};

/* Subroutine */ int cspt01_(char *uplo, integer *n, complex *a, complex *
	afac, integer *ipiv, complex *c__, integer *ldc, real *rwork, real *
	resid)
{
    /* System generated locals */
    integer c_dim1, c_offset, i__1, i__2, i__3, i__4, i__5;
    complex q__1;

    /* Local variables */
    integer i__, j, jc;
    real eps;
    integer info;
    extern logical lsame_(char *, char *);
    real anorm;
    extern doublereal slamch_(char *);
    extern /* Subroutine */ int claset_(char *, integer *, integer *, complex
	    *, complex *, complex *, integer *);
    extern doublereal clansp_(char *, char *, integer *, complex *, real *);
    extern /* Subroutine */ int clavsp_(char *, char *, char *, integer *,
	    integer *, complex *, integer *, complex *, integer *, integer *);
    extern doublereal clansy_(char *, char *, integer *, complex *, integer *,
	     real *);

/*  -- LAPACK test routine (version 3.1) -- */
/*     Univ. of Tennessee, Univ. of California Berkeley and NAG Ltd.. */
/*     November 2006 */

/*     .. Scalar Arguments .. */
/*     .. */
/*     .. Array Arguments .. */
/*     .. */

/*  Purpose */
/*  ======= */

/*  CSPT01 reconstructs a symmetric indefinite packed matrix A from its */
/*  diagonal pivoting factorization A = U*D*U' or A = L*D*L' and computes */
/*  the residual */
/*     norm( C - A ) / ( N * norm(A) * EPS ), */
/*  where C is the reconstructed matrix and EPS is the machine epsilon. */

/*  Arguments */
/*  ========== */

/*  UPLO    (input) CHARACTER*1 */
/*          Specifies whether the upper or lower triangular part of the */
/*          Hermitian matrix A is stored: */
/*          = 'U':  Upper triangular */
/*          = 'L':  Lower triangular */

/*  N       (input) INTEGER */
/*          The order of the matrix A.  N >= 0. */

/*  A       (input) COMPLEX array, dimension (N*(N+1)/2) */
/*          The original symmetric matrix A, stored as a packed */
/*          triangular matrix. */

/*  AFAC    (input) COMPLEX array, dimension (N*(N+1)/2) */
/*          The factored form of the matrix A, stored as a packed */
/*          triangular matrix.  AFAC contains the block diagonal matrix D */
/*          and the multipliers used to obtain the factor L or U from the */
/*          L*D*L' or U*D*U' factorization as computed by CSPTRF. */

/*  IPIV    (input) INTEGER array, dimension (N) */
/*          The pivot indices from CSPTRF. */

/*  C       (workspace) COMPLEX array, dimension (LDC,N) */

/*  LDC     (integer) INTEGER */
/*          The leading dimension of the array C.  LDC >= max(1,N). */

/*  RWORK   (workspace) REAL array, dimension (N) */

/*  RESID   (output) REAL */
/*          If UPLO = 'L', norm(L*D*L' - A) / ( N * norm(A) * EPS ) */
/*          If UPLO = 'U', norm(U*D*U' - A) / ( N * norm(A) * EPS ) */

/*  ===================================================================== */

/*     .. Parameters .. */
/*     .. */
/*     .. Local Scalars .. */
/*     .. */
/*     .. External Functions .. */
/*     .. */
/*     .. External Subroutines .. */
/*     .. */
/*     .. Intrinsic Functions .. */
/*     .. */
/*     .. Executable Statements .. */

/*     Quick exit if N = 0. */

    /* Parameter adjustments */
    --a;
    --afac;
    --ipiv;
    c_dim1 = *ldc;
    c_offset = 1 + c_dim1;
    c__ -= c_offset;
    --rwork;

    /* Function Body */
    if (*n <= 0) {
	*resid = 0.f;
	return 0;
    }

/*     Determine EPS and the norm of A. */

    eps = slamch_("Epsilon");
    anorm = clansp_("1", uplo, n, &a[1], &rwork[1]);

/*     Initialize C to the identity matrix. */

    claset_("Full", n, n, &c_b1, &c_b2, &c__[c_offset], ldc);

/*     Call CLAVSP to form the product D * U' (or D * L' ). */

    clavsp_(uplo, "Transpose", "Non-unit", n, n, &afac[1], &ipiv[1], &c__[
	    c_offset], ldc, &info);

/*     Call CLAVSP again to multiply by U ( or L ). */

    clavsp_(uplo, "No transpose", "Unit", n, n, &afac[1], &ipiv[1], &c__[
	    c_offset], ldc, &info);

/*     Compute the difference  C - A . */

    if (lsame_(uplo, "U")) {
	jc = 0;
	i__1 = *n;
	for (j = 1; j <= i__1; ++j) {
	    i__2 = j;
	    for (i__ = 1; i__ <= i__2; ++i__) {
		i__3 = i__ + j * c_dim1;
		i__4 = i__ + j * c_dim1;
		i__5 = jc + i__;
		q__1.r = c__[i__4].r - a[i__5].r, q__1.i = c__[i__4].i - a[
			i__5].i;
		c__[i__3].r = q__1.r, c__[i__3].i = q__1.i;
/* L10: */
	    }
	    jc += j;
/* L20: */
	}
    } else {
	jc = 1;
	i__1 = *n;
	for (j = 1; j <= i__1; ++j) {
	    i__2 = *n;
	    for (i__ = j; i__ <= i__2; ++i__) {
		i__3 = i__ + j * c_dim1;
		i__4 = i__ + j * c_dim1;
		i__5 = jc + i__ - j;
		q__1.r = c__[i__4].r - a[i__5].r, q__1.i = c__[i__4].i - a[
			i__5].i;
		c__[i__3].r = q__1.r, c__[i__3].i = q__1.i;
/* L30: */
	    }
	    jc = jc + *n - j + 1;
/* L40: */
	}
    }

/*     Compute norm( C - A ) / ( N * norm(A) * EPS ) */

    *resid = clansy_("1", uplo, n, &c__[c_offset], ldc, &rwork[1]);

    if (anorm <= 0.f) {
	if (*resid != 0.f) {
	    *resid = 1.f / eps;
	}
    } else {
	*resid = *resid / (real) (*n) / anorm / eps;
    }

    return 0;

/*     End of CSPT01 */

} /* cspt01_ */

{'repo_name': 'FangGet/ORB_SLAM2_Android', 'stars': '247', 'repo_language': 'C++', 'file_name': 'org.eclipse.jdt.core.prefs', 'mime_type': 'text/plain', 'hash': 8300161065690813501, 'source_dataset': 'data'}


# End-to-end Reusable ML Pipeline with Seldon and Kubeflow

In this example we showcase how to build re-usable components to build an ML pipeline that can be trained and deployed at scale.

We will automate content moderation on the Reddit comments in /r/science building a machine learning NLP model with the following components:

This tutorial will break down in the following sections:

1) Run all the services (Kubeflow and Seldon)

2) Test and build all our reusable pipeline steps

3) Use Kubeflow to Train the Pipeline and Deploy to Seldon

5) Test Seldon Deployed ML REST Endpoints

6) Visualise Seldon's Production ML Pipelines

Make sure you install the following dependencies, as they are critical for this example to work:

* A Kubernetes cluster running v1.13 or above (minkube / docker-for-windows work well if enough RAM)

* kfctl 0.5.1 - Please use this exact version as there are major changes every few months

* Python DEV requirements (we'll install them below)

Let's get started! 🚀🔥 We will be building the end-to-end pipeline below:

stdout

python-dateutil

kubernetes

click

numpy

## 1) Run all the services (Kubeflow and Seldon)

Kubeflow's CLI allows us to create a project which will allow us to build the configuration we need to deploy our kubeflow and seldon clusters.

app.yaml

!kfctl init kubeflow-seldon
!ls kubeflow-seldon

Now we run the following commands to basically launch our Kubeflow cluster with all its components.

It may take a while to download all the images for Kubeflow so feel free to make yourself a cup of ☕.

If you have a terminal you can see how the containers are created in real-time by running `kubectl get pods -n kubeflow -w`.

%%bash
cd kubeflow-seldon
kfctl generate all -V
kfctl apply all -V

For this we'll need Helm to be running, so we'll initialise it.

$HELM_HOME has been configured at /home/alejandro/.helm.

Warning: Tiller is already installed in the cluster.

(Use --client-only to suppress this message, or --upgrade to upgrade Tiller to the current version.)

Happy Helming!

deployment "tiller-deploy" successfully rolled out

kubectl rollout status deploy/tiller-deploy -n kube-system

Once it's running we can now run the installation command for Seldon.

As you can see, we are running the Seldon Operator in the Kubeflow namespace.

NAME:   old-mastiff

LAST DEPLOYED: Tue May 28 11:47:35 2019

NAMESPACE: kubeflow

STATUS: DEPLOYED

RESOURCES:

==> v1/ClusterRole

NAME                          AGE

seldon-operator-manager-role  0s

==> v1/ClusterRoleBinding

NAME                                 AGE

seldon-operator-manager-rolebinding  0s

==> v1/Pod(related)

NAME                                  READY  STATUS             RESTARTS  AGE

seldon-operator-controller-manager-0  0/1    ContainerCreating  0         0s

==> v1/Secret

NAME                                   TYPE    DATA  AGE

seldon-operator-webhook-server-secret  Opaque  0     0s

==> v1/Service

NAME                                        TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)  AGE

seldon-operator-controller-manager-service  ClusterIP  10.101.228.36  <none>       443/TCP  0s

==> v1/StatefulSet

NAME                                READY  AGE

seldon-operator-controller-manager  0/1    0s

==> v1beta1/CustomResourceDefinition

NAME                                         AGE

seldondeployments.machinelearning.seldon.io  0s

NOTES:

NOTES: TODO

!helm install seldon-core-operator --namespace kubeflow --repo https://storage.googleapis.com/seldon-charts

Check all the Seldon Deployment is running

seldon-operator-controller-manager-0                       1/1     Running   1          6s

!kubectl get pod -n kubeflow | grep seldon

At the time of writing we need to make some updates in the Argo images with the following commands below.

(This basically changes the images to the latest ones, otherwise we will get an error when we attach the volume)

deployment.extensions/workflow-controller patched

deployment.extensions/ml-pipeline patched

!kubectl -n kubeflow patch deployments. workflow-controller --patch '{"spec": {"template": {"spec": {"containers": [{"name": "workflow-controller", "image": "argoproj/workflow-controller:v2.3.0-rc3"}]}}}}'

!kubectl -n kubeflow patch deployments. ml-pipeline --patch '{"spec": {"template": {"spec": {"containers": [{"name": "ml-pipeline-api-server", "image": "elikatsis/ml-pipeline-api-server:0.1.18-pick-1289"}]}}}}'

# !kubectl -n kubeflow patch configmaps workflow-controller-configmap --patch '{"data": {"config": "{ executorImage: argoproj/argoexec:v2.3.0-rc3,artifactRepository:{s3: {bucket: mlpipeline,keyPrefix: artifacts,endpoint: minio-service.kubeflow:9000,insecure: true,accessKeySecret: {name: mlpipeline-minio-artifact,key: accesskey},secretKeySecret: {name: mlpipeline-minio-artifact,key: secretkey}}}}" }}'

The last command you need to run actually needs to be manual as the patch cannot change configmap contents directly

You need to run the edit commad and change the executorImage to: `argoproj/argoexec:v2.3.0-rc3`

The command should be run from a terminal:

kubectl edit configmaps workflow-controller-configmap -n kubeflow

## 2) Test and build all our reusable pipeline steps

We will start by building each of the components in our ML pipeline.

### Let's first have a look at our clean_text step:

clean_text	 lr_text_classifier  tfidf_vectorizer

data_downloader  spacy_tokenize

Like in this step, all of the other steps can be found in the `pipeline/pipeline_steps/` folder, and all have the following structure:

* `pipeline_step.py` which exposes the functionality through a CLI

* `Transformer.py` which transforms the data accordingly

* `requirements.txt` which states the python dependencies to run

* `build_image.sh` which uses `s2i` to build the image with one line

### Let's check out the CLI for clean_text

The pipeline_step CLI is the entry point for the kubeflow image as it will be able to pass any relevant parameters

Usage: pipeline_step.py [OPTIONS]

Options:

--in-path TEXT

--out-path TEXT

--help           Show this message and exit.

This is actually a very simple file, as we are using the click library to define the commands:

import dill

import click

import dill

try:

# Running for tests

from .Transformer import Transformer

except:

# Running from CLI

from Transformer import Transformer

@click.command()

@click.option('--in-path', default="/mnt/raw_text.data")

@click.option('--out-path', default="/mnt/clean_text.data")

def run_pipeline(in_path, out_path):

clean_text_transformer = Transformer()

with open(in_path, 'rb') as in_f:

x = dill.load(in_f)

y = clean_text_transformer.predict(x)

with open(out_path, "wb") as out_f:

dill.dump(y, out_f)

if __name__ == "__main__":

The Transformer is where the data munging and transformation stage comes in, which will be wrapped by the container and exposed through the Seldon Engine to ensure our pipeline can be used in production.

Seldon provides multiple different features, such as abilities to send custom metrics, pre-process / post-process data and more. In this example we will only be exposing the `predict` step.

import re

from html.parser import HTMLParser

import numpy as np

import logging

class Transformer():

__html_parser = HTMLParser()

__uplus_pattern = \

re.compile("\<[uU]\+(?P<digit>[a-zA-Z0-9]+)\>")

__markup_link_pattern = \

re.compile("\[(.*)\]\((.*)\)")

def predict(self, X, feature_names=[]):

logging.warning(X)

f = np.vectorize(Transformer.transform_clean_text)

X_clean = f(X)

return X_clean

def fit(self, X, y=None, **fit_params):

return self

@staticmethod

def transform_clean_text(raw_text):

try:

decoded = raw_text.encode("ISO-8859-1").decode("utf-8")

except:

decoded = raw_text.encode("ISO-8859-1").decode("cp1252")

html_unescaped =Transformer.\

html_unescaped = re.sub(r"\r\n", " ", html_unescaped)

html_unescaped = re.sub(r"\r\r\n", " ", html_unescaped)

html_unescaped = re.sub(r"\r", " ", html_unescaped)

html_unescaped = html_unescaped.replace(">", " > ")

html_unescaped = html_unescaped.replace("<", " < ")

html_unescaped = html_unescaped.replace("--", " - ")

html_unescaped = Transformer.__uplus_pattern.sub(

" U\g<digit> ", html_unescaped)

html_unescaped = Transformer.__markup_link_pattern.sub(

" \1 \2 ", html_unescaped)

html_unescaped = html_unescaped.replace("\\", "")

return html_unescaped

If you want to understand how the CLI pipeline talks to each other, have a look at the end to end test in `pipeline/pipeline_tests/`:

============================= test session starts ==============================

platform linux -- Python 3.7.3, pytest-4.5.0, py-1.8.0, pluggy-0.11.0

rootdir: /home/alejandro/Programming/kubernetes/seldon/seldon-core/examples/kubeflow

collected 1 item

pipeline/pipeline_tests/test_pipeline.py .                               [100%]

===================== 1 passed, 9 warnings in 3.26 seconds =====================

To build the image we provide a build script in each of the steps that contains the instructions:

#!/bin/bash

s2i build . seldonio/seldon-core-s2i-python3:0.6 clean_text_transformer:0.1

The only thing you need to make sure is that Seldon knows how to wrap the right model and file.

This can be achieved with the s2i/environment file.

As you can see, here we just tell it we want it to use our `Transformer.py` file:

Once this is defined, the only thing we need to do is to run the `build_image.sh` for all the reusable components.

cd pipeline/pipeline_steps/clean_text/ && ./build_image.sh

cd ../data_downloader && ./build_image.sh

cd ../lr_text_classifier && ./build_image.sh

cd ../spacy_tokenize && ./build_image.sh

cd ../tfidf_vectorizer && ./build_image.sh

## 3) Train our NLP Pipeline through the Kubeflow UI

We can access the Kubeflow dashboard to train our ML pipeline via http://localhost/_/pipeline-dashboard

If you can't edit this, you need to make sure that the ambassador gateway service is accessible:

NAME         TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE

ambassador   NodePort   10.97.236.196   <none>        80:30209/TCP   8m58s

In my case, I need to change the kind from `NodePort` into `LoadBalancer` which can be done with the following command:

service/ambassador patched

!kubectl patch svc ambassador --type='json' -p '[{"op":"replace","path":"/spec/type","value":"LoadBalancer"}]' -n kubeflow

Now that I've changed it to a loadbalancer, it has allocated the external IP as my localhost so I can access it at http://localhost/_/pipeline-dashboard

NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE

ambassador   LoadBalancer   10.97.236.196   localhost     80:30209/TCP   9m20s

If this was successfull, you should be able to access the dashboard

Now we want to generate the pipeline. For this we can use the DSL provided by kubeflow to define the actual steps required.

import kfp.dsl as dsl

import yaml

from kubernetes import client as k8s

@dsl.pipeline(

name='NLP',

description='A pipeline demonstrating reproducible steps for NLP'

def nlp_pipeline(

lr_prediction_path='/mnt/prediction.data',

Pipeline

vop = dsl.VolumeOp(

name='my-pvc',

modes=["ReadWriteMany"],

download_step = dsl.ContainerOp(

command="python",

arguments=[

"--labels-path", labels_path,

"--features-path", raw_text_path,

"--csv-url", csv_url,

"--csv-encoding", csv_encoding,

"--features-column", features_column,

"--labels-column", labels_column

pvolumes={"/mnt": vop.volume}

clean_step = dsl.ContainerOp(

command="python",

arguments=[

"--in-path", raw_text_path,

"--out-path", clean_text_path,

pvolumes={"/mnt": download_step.pvolume}

tokenize_step = dsl.ContainerOp(

name='tokenize',

command="python",

arguments=[

"--in-path", clean_text_path,

"--out-path", spacy_tokens_path,

pvolumes={"/mnt": clean_step.pvolume}

vectorize_step = dsl.ContainerOp(

name='vectorize',

command="python",

arguments=[

"--in-path", spacy_tokens_path,

"--out-path", tfidf_vectors_path,

"--max-features", tfidf_max_features,

"--ngram-range", tfidf_ngram_range,

"--action", "train",

"--model-path", tfidf_model_path,

pvolumes={"/mnt": tokenize_step.pvolume}

predict_step = dsl.ContainerOp(

name='predictor',

command="python",

arguments=[

"--in-path", tfidf_vectors_path,

"--labels-path", labels_path,

"--out-path", lr_prediction_path,

"--c-param", lr_c_param,

"--action", "train",

"--model-path", lr_model_path,

pvolumes={"/mnt": vectorize_step.pvolume}

try:

seldon_config = yaml.load(open("../deploy_pipeline/seldon_production_pipeline.yaml"))

except:

# If this file is run from the project core directory

seldon_config = yaml.load(open("deploy_pipeline/seldon_production_pipeline.yaml"))

deploy_step = dsl.ResourceOp(

name="seldondeploy",

attribute_outputs={"name": "{.metadata.name}"})

if __name__ == '__main__':

import kfp.compiler as compiler

compiler.Compiler().compile(nlp_pipeline, __file__ + '.tar.gz')

As you can see in the DSL, we have the ContainerOp - each of those is a step in the Kubeflow pipeline.

At the end we can see the `seldondeploy` step which basically deploys the trained pipeline

The definition of the SeldonDeployment graph is provided in the `deploy_pipeline/seldon_production_pipeline.yaml` file.

The seldondeployment file defines our production execution graph using the same reusable components.

apiVersion: machinelearning.seldon.io/v1alpha2

kind: SeldonDeployment

metadata:

labels:

app: seldon

name: "seldon-deployment-{{workflow.name}}"

namespace: kubeflow

spec:

annotations:

project_name: NLP Pipeline

deployment_version: v1

name: "seldon-deployment-{{workflow.name}}"

oauth_key: oauth-key

oauth_secret: oauth-secret

predictors:

- componentSpecs:

- spec:

containers:

- image: clean_text_transformer:0.1

imagePullPolicy: IfNotPresent

name: cleantext

resources:

requests:

memory: 1Mi

- image: spacy_tokenizer:0.1

imagePullPolicy: IfNotPresent

name: spacytokenizer

- image: tfidf_vectorizer:0.1

imagePullPolicy: IfNotPresent

name: tfidfvectorizer

volumeMounts:

- name: mypvc

mountPath: /mnt

- image: lr_text_classifier:0.1

imagePullPolicy: IfNotPresent

name: lrclassifier

volumeMounts:

- name: mypvc

mountPath: /mnt

terminationGracePeriodSeconds: 20

volumes:

- name: mypvc

persistentVolumeClaim:

claimName: "{{workflow.name}}-my-pvc"

graph:

children:

- name: spacytokenizer

endpoint:

type: REST

type: MODEL

children:

- name: tfidfvectorizer

endpoint:

type: REST

type: MODEL

children:

- name: lrclassifier

endpoint:

type: REST

type: MODEL

children: []

name: cleantext

endpoint:

type: REST

type: MODEL

name: single-model

replicas: 1

annotations:

predictor_version: v1

If we look at the file we'll be using to deploy our pipeline, we can see that it has the following key points:

1) Reusable components definitions as containerSpecs: cleantext, spacytokenizer, tfidfvectorizer & lrclassifier

2) DAG (directed acyclic graph) definition for REST pipeline: cleantext -> spacytokenizer -> tfidfvectorizer -> lrclassifier

This graph in our production deployment looks as follows:

### Generate the pipeline files to upload to Kubeflow

To generate the pipeline we just have to run the pipeline file, which will output the `tar.gz` file that will be uploaded.

%%bash
# Generating graph definition
python train_pipeline/nlp_pipeline.py
ls train_pipeline/

We now need to upload the resulting `nlp_pipeline.py.tar.gz` file generated.

This can be done through the "Upload PIpeline" button in the UI at http://localhost/_/pipeline-dashboard.

Once it's uploaded, we want to create and trigger a run! You should now be able to see how each step is executed:

### Inspecting the data created in the Persistent Volume

The pipeline saves the output of the pipeline together with the trained model in the persistent volume claim.

The persistent volume claim is the same name as the argo workflow:

NAME        AGE
nlp-bddff   2m

Our workflow is there! So we can actually access it by running

nlp-bddff

!kubectl get workflow -n kubeflow -o jsonpath='{.items[0].metadata.name}'

And we can use good old `sed` to insert this workflow name in our PVC-Access controler which we can use to inspect the contents of the volume:

apiVersion: v1

kind: Pod

metadata:

name: pvc-access-container

spec:

containers:

- name: pvc-access-container

image: busybox

command: ["/bin/sh", "-ec", "sleep 1000"]

volumeMounts:

- name: mypvc

mountPath: /mnt

volumes:

- name: mypvc

persistentVolumeClaim:

claimName: nlp-b7qt8-my-pvc

We just need to apply this container with our kubectl command, and we can use it to inspect the mounted folder:

pod/pvc-access-container created

!sed "s/PVC_NAME/"$(kubectl get workflow -n kubeflow -o jsonpath='{.items[0].metadata.name}')"-my-pvc/g" deploy_pipeline/pvc-access.yaml | kubectl -n kubeflow apply -f -

NAME                   READY   STATUS    RESTARTS   AGE

pvc-access-container   1/1     Running   0          6s

!kubectl get pods -n kubeflow pvc-access-container

Now we can run an `ls` command to see what's inside:

clean.data       lr.model         text.data        tfidf.model

labels.data      prediction.data  tfidf.data       tokens.data

!kubectl -n kubeflow exec -it pvc-access-container ls /mnt

pod "pvc-access-container" deleted

!kubectl delete -f deploy_pipeline/pvc-access.yaml -n kubeflow

Now that it's running we have a production ML text pipeline that we can Query using REST and GRPC

First we can check if our Seldon deployment is running with

NAME                          AGE
seldon-deployment-nlp-b7qt8   57m

!kubectl -n kubeflow get seldondeployment

We will need the Seldon Pipeline Deployment name to reach the API, so we can get it using:

!kubectl -n kubeflow get seldondeployment -o jsonpath='{.items[0].metadata.name}'

Now we can interact with our API in two ways:

1) Using CURL or any client like PostMan

When using CURL, the only thing we need to provide is the data in JSON format, as well as the url, which is of the format:

"meta": {

"puid": "k89krp6t7tfgb386nt6vc3iftk",

"tags": {

"routing": {

"cleantext": -1,

"tfidfvectorizer": -1,

"spacytokenizer": -1

"requestPath": {

"cleantext": "clean_text_transformer:0.1",

"tfidfvectorizer": "tfidf_vectorizer:0.1",

"lrclassifier": "lr_text_classifier:0.1",

"spacytokenizer": "spacy_tokenizer:0.1"

"metrics": []

"data": {

"names": ["t:0", "t:1"],

"ndarray": [[0.6729318752883149, 0.3270681247116851]]

stderr

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

Dload  Upload   Total   Spent    Left  Speed

0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100    72    0     0  100    72      0    354 --:--:-- --:--:-- --:--:--   356

100   599  100   527  100    72    516     70  0:00:01  0:00:01 --:--:--   588

curl -X POST -H 'Content-Type: application/json' \

-d "{'data': {'names': ['text'], 'ndarray': ['Hello world this is a test']}}" \

http://127.0.0.1/seldon/kubeflow/$(kubectl -n kubeflow get seldondeployment -o jsonpath='{.items[0].metadata.name}')/api/v0.1/predictions

We can also use the Python SeldonClient to interact with the pipeline we just deployed

stdout

Success:True message:

Request:

data {

names: "text"

ndarray {

values {

string_value: "Hello world this is a test"

Response:

meta {

puid: "qtdca40d3s0463nn4ginhkvc6t"

routing {

key: "cleantext"

value: -1

routing {

key: "spacytokenizer"

value: -1

routing {

key: "tfidfvectorizer"

value: -1

requestPath {

key: "cleantext"

value: "clean_text_transformer:0.1"

requestPath {

key: "lrclassifier"

value: "lr_text_classifier:0.1"

requestPath {

key: "spacytokenizer"

value: "spacy_tokenizer:0.1"

requestPath {

key: "tfidfvectorizer"

value: "tfidf_vectorizer:0.1"

data {

names: "t:0"

names: "t:1"

ndarray {

values {

list_value {

values {

number_value: 0.6729318752883149

values {

number_value: 0.3270681247116851

from seldon_core.seldon_client import SeldonClient

port = "80" # Make sure you use the port above

batch = np.array(["Hello world this is a test"])

deployment_name = subprocess.getoutput("kubectl -n kubeflow get seldondeployment -o jsonpath='{.items[0].metadata.name}'")

## 6) Visualise Seldon's Production ML Pipelines

We can visualise the performance using the SeldonAnalytics package, which we can deploy using:

!helm install seldon-core-analytics --repo https://storage.googleapis.com/seldon-charts --namespace kubeflow

In my case, similar to what I did with Ambassador, I need to make sure the the service is a LoadBalancer instead of a NodePort

service/grafana-prom patched

!kubectl patch svc grafana-prom --type='json' -p '[{"op":"replace","path":"/spec/type","value":"LoadBalancer"}]' -n kubeflow

NAME           TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE

grafana-prom   LoadBalancer   10.98.248.223   localhost     80:32445/TCP   64m

!kubectl get svc grafana-prom -n kubeflow

Now we can access it at the port provided, in my case it is http://localhost:32445/d/3swM2iGWz/prediction-analytics?refresh=5s&orgId=1

(initial username is admin and password is password, which will be requested to be changed on the first login)

Generate a bunch of requests and visualise:

## You now have a full end-to-end training and production NLP pipeline 😎

conda-env-reddit-classification-py

Python [conda env:reddit-classification] *

{'repo_name': 'PipelineAI/pipeline', 'stars': '4017', 'repo_language': 'Jsonnet', 'file_name': 'hdfs_test.py', 'mime_type': 'text/plain', 'hash': -1876631940125458232, 'source_dataset': 'data'}