(Scott Daniel) Feature/CATSIM-136 cosmology package
This branch adds a wrapper class CosmologyWrapper which provides a controlled access to the astropy.cosmology methods we will use most often (it should be easily extensible if we need to make more methods available; there is also the option for the user to access astropy.cosmology directly)

An example mixin using these methods is available in lsst.sims.photUtils.examples.CosmologyMixin.py

During the course of this work, I ended up taking a large chunk of code out of testPhotometry.py and moving it into lsst.sims.photUtils.utils.testUtils.py  These were all methods that generated test databases and catalogs that I thought might be useful for unit tests beyond testPhotometry.py (indeed, I ended up using one of them in testCosmology.py, which is why I moved the code in the first place).

* added booleans to determine if dark energy is a cosmological constant
and if the universe is flat
* test that exceptions get raised if you try to calculate something
before initializing the cosmology
* force distance moduli > 0.0
* added unit test to make sure distance modulus at z=0 is handled properly
turned out, I was using numpy.where() wrong in the actual distanceModulus
method, so I fixed that
* changed naming convention of get_ and set_ methods
* changed the order of API determining if statements
* added comments to CosmologyWrapper.py
* moved methods to calculate omega and comoving distance for control purposes
to testUtils.py
* added comments to unit test
* added comment to CosmologyMixin.py


(Scott Daniel) I apologize about the appearance of this diff.  It looks as if stash was smart enough to realize I copy-and-pasted this from testPhotometry.py, but not smart enough to realize it should treat this as a totally separate file.


(Simon Krughoff) Please document the arguments using the standard @param, @return notation.


(Simon Krughoff) Should document all parameters here too.


(Simon Krughoff) This must be a typo.  self.hunits -> self.hUnits.


(Simon Krughoff) I don't understand why the units are hard coded.  Can't you do:
self.hUnits = H.unit
?
Is it worth hard coding units for older versions of the package?


(Simon Krughoff) Why is this not just a constructor with default arguments?  I must be missing a reason why you don't want to just construct a CosmologyWrapper object.


(Simon Krughoff) Can you explain a little about why you have to convert units?  Shouldn't you just be able to pass through the value and return the result in whatever units make sense?


(Simon Krughoff) If you make CosmologyWrapper class with a constructor with defaulted parameters, you could eliminate all of these error checks since they are not classmethods.


(Simon Krughoff) For all these wrapper functions, you can probably use the @wraps decorator to pass on the astropy.cosmology docstrings.


Just noting in passing that there's something confusing about how this note is phrased, which made me do a double take. I'd rewrite it in a way more like "Unlike KFold(n_folds=n // p), which creates exactly `n // p` non-overlapping test sets, `LeavePOut(p=p)` creates all (n choose p) overlapping test sets.

This would allow the next paragraph to be simplified too.


(Simon Krughoff) I'd like to see if it's possible to forward docstrings from the cosmology module and I'd like to know why CosmolgyWrapper can't be a standard class with a constructor (Is it a mixin issue?).  Otherwise, it looks good.


(Scott Daniel) Good point.  When I started writing it, I envisioned CosmologyWrapper working as a mixin.  That no longer makes sense (we can just write mixins in the style of ExampleCosmologyMixin that incorporate a CosmologyWrapper object).  I will re-factor this as a proper object.


(Scott Daniel) Ultimately, yes.

astropy.cosmology (at least, the newer version; the version that ships with Anaconda does not contain the units functionality) allows you to convert results to whatever units you want.  However, you do that by performing the calculation, which returns H (or comoving_distance or whatever) in the default units, and then converting to your desired units.  You convert to the desired units by instantiating an astropy.units object and telling astropy to convert H to units that comport with that instantiation.  Creating an astropy.units instantiation takes time.  Therefore, when CosmologyWrapper is first initialized, we set a bunch of member variables storing the units in which we want everything to be output.  When we calculate H, we check that H came out in the desired units and convert if necessary.

This process is in place to guard against the possibility that the developers of astropy.cosmology will change the default units of various calculations at sometime in the future (i.e. returning H in inverse seconds or somesuch).

Does this answer your question?


(Scott Daniel) The idea is that out downstream code (i.e. ExampleCosmologyMixin) will be expecting H and comoving_distance in specific units.  I am hard-coding in what we expect the output to look like, just in case astropy.cosmology decides to change its default output units in some future version.

so: self.hUnits is not a variable describing what we think the output will be, it is a variable we are setting to force H to come out in km /s /Mpc, because that is what our code expects when you calculate H.


(Scott Daniel) yup. fixed


hey, so not a show stopper for me but i am still wondering why we are maintaining uat-core though it does not exist and is never referred to. I mean, *if* we have a library with that name ever that is in actual use i totally support it. no biggie and no reply needed just want to get this on the record but feel free to reply if you'd like :)


(Scott Daniel) done


(Scott Daniel) It does work.  My one concern is that it pulls in the astropy.cosmology docstrings and completely overwrites ours.

This means that if they ever change the default units output by their methods, the docstrings will claim that the output is in their units, rather than ours.  This doesn't seem like it is likely to happen.  But I am generally paranoid.

What if I added notes to the end of our docstrings to the effect that 'this wraps the astropy.cosmology method ____ and forces the units to be ___' so that interested users could look up the method in question?


I <3 Navmesh


(Scott Daniel) Feature/CATSIM-154 scons testreadgalfast
Because of the way the testReadGalfast.py unit test was implemented, SCons was not actually running the test.  This branch fixes that.


(Yusra AlSayyad) Sorry I didn't see this until now. I'll take a look today.


(Yusra AlSayyad) Not worth changing here, but in general lsst imports are grouped together.


(Lynne Jones) Bugfix/SIM-1060 new astropy fails cosmology
modified get_cosmologicalDistanceModulus so that it will automatically
return an empty numpy array if redshift comes back as an empty
numpy array (astropy was complaining whenever InstanceCatalog attempted
to verify its column requirements)


(Bryce Kalmbach) Bugfix/CATSIM-119 improve speed of readgalfast and
Finished the selectStarSED speedup and adjusted unittests for the new routine as well.


(Scott Daniel) looks good to merge


(Scott Daniel) Feature/SIM-1046 refactor photometric uncertainty
This pull request changes the way that photUtils handles both the photometric uncertainties and 5-sigma limiting magnitude (m5) on which they depend. I have tried to streamline the process so that one can just pass magnitudes into the uncertainty calculation methods and get results out.

* gets rid of calculatePhotometricUncertaintyFromColumnNames
* did away with the 'uRecalc' naming conventions for total galaxy
magnitudes.  They are now called 'lsst_u' etc.  This necessitated
splitting the Variability mixin into VariabilityStars and VariabilityGalaxies,
since they now both have 'lsst_u' as a column.
* test photometric catalogs of galaxies in testSetupRoutines.py
* made changes to testPhotometry.py to reflect the fact that galaxy magnitudes
now get padded with NaNs rather than Nones
* reformatted testSetupRoutines.py
* refactored uncertainty calculation so that it relies on a free-standing
method in noiseUtilities.py
* removed some redundant variables from unittest
* added line to testPhotometry.py to test calcSNR_gamma
* added tests to make sure calcSNR_gamma raises exceptions when it should
* updated docstrings


(Bryce Kalmbach) Hi @danielsf,
I'm getting all of the unit tests failing with a message similar to "AttributeError: 'testStars' object has no attribute 'all_calculated_columns'".


(Scott Daniel) are you on master of sims_catalogs_measures?  That was a feature I added to InstanceCatalog during the hack week in Tucson


(Bryce Kalmbach) Ah, yep git pull on that fixed it. Thanks, I'll try to get through this review tonight, sorry it took so long to get to it.


(Bryce Kalmbach) typo "the the"


(Bryce Kalmbach) Could you add the name of the output (I see 'snr') and something about it's type and shape?


(Bryce Kalmbach) to compare testGalaxyCatalog not testStarCatalog?


(Bryce Kalmbach) Bugfix/CATSIM-153 speed up readgalfast unit test
* Amended class TestSelectStarSED to run faster. Fixed very minor grammar mistake in selectStarSed.py
* Amended testReadGalfast to ask for different directories of SEDs when testing loadGalfast. Made changes in loadGalfast to make this possible.
* Amended class TestSelectStarSED to run faster. Fixed very minor grammar mistake in selectStarSed.py
* Amended testReadGalfast to ask for different directories of SEDs when testing loadGalfast. Made changes in loadGalfast to make this possible.
* Made little change from Yusra's comment on CATSIM-154 pull request
* Made changes in response to buildbot failure


(Scott Daniel) I think we should make these assignments in the selectStarSED class (maybe just make kuruczDir, mltDir, etc. kwargs of it's __init__() method).  That way, if we ever find a reason to use selectStarSED without using readGalfast, the infrastructure will be in place to redirect the SED directories.


(Scott Daniel) Should this be in setUpClass (which is called once for every block of tests in this particular unit test class), as opposed to setUp which is called once for every individual test in this particular unit test class?


(Scott Daniel) Similarly, should this be in tearDownClass, rather than tearDown?


(Scott Daniel) again, I think kuruczDir should be a kwarg of the selectStarSED __init__()


(Scott Daniel) again: mltDir should be a kwarg of selectStarSED's __init__()


(Scott Daniel) this is me testing to see if I can actually print the underscores in init():  \code__init__()


(Scott Daniel) \__init__()


(Scott Daniel) Whatever method ends up establishing the testReadGalfastSpectra directory tree should declare some variable like self.testKuruczDir='testReadGalfastSpectra/starSED/kurucz' which you can pass to os.listdir here.  That way, if we ever need to change the directory name, we only have to do it in one place.


(Scott Daniel) same comment as with the testReadGalfastSpectra/starSED/kurucz directory name (store it in a class or class member variable)


(Scott Daniel) again: class variable


(Scott Daniel) Here is another place to apply the changes to how directory names are stored(store the testReadGalfast directory structure in a member variable of the unit test class; pass it to selectStarSED as a kwarg of \__init__() )


(Scott Daniel) Why is this not in setUpClass?


(Scott Daniel) if you take my advice about moving the testReadGalfastSpectra set up code to setUpClass, this should be in tearDownClass


(Scott Daniel) Looks good.  My only comments are that it will be easier to up-keep the code if you store the dummy test directory structure in a class member variable associated with the unit test classes.  I also think that the dummy directories should be set up and torn down by setUpClass and tearDownClass, rather than setUp and tearDown.  Generally, setUpClass and tearDownClass are called less often than setUp and tearDown, so that might make this a slightly more efficient test (unless there is some consideration I neglected).


(Scott Daniel) Or I can just do the review now.

It looks good.  There is one comment I have (I made it on the commit, rather than the pull request, so that I could be sure to see only what changed since the last iteration): I think the kuruczDir, mltDir, and wdDir kwargs in selectStarSED() should be relative to self.SEDDir.

Feel free to push back if you disagree.

Other than that, this is ready to merge.


(Scott Daniel) Feature/CATSIM-138 cosmological magnitudes
This branch adapts the CosmologyObject (formerly CosmologyWrapper) class into a mixin that can be incorporated into InstanceCatalog daughter classes to add a cosmological distance module column and add that distance modulus to galaxies' component magnitudes.

Note: As a part of this work, I modified the applyAvAndRedshift method in PhotometryBase so that it no longer includes cosmological dimming, since that would be redundant with the cosmological distance modulus and the cosmological dimming as implemented by the SED class is just a renormalization (which, if present, the magNorm database variable ought to take account of).

* renamed CosmologyWrapper.py CosmologyObject.py
* applyAvAndRedshift in Photometry.py no longer applies cosmological
dimming.  The rationale is that the galaxy's magNorm accounts for
any dimming necessary to take the galaxy down to its apparent
magnitude.
This change implies a change in the example CosmologyMixin, which no longer
has to worry about undoing the cosmological dimming applied by
applyAvAndRedshift
* added CosmologyWrapper class to provide a self.cosmology member variable
to InstanceCatalog daughter classes.  This also adds a distanceModulus column
to the database.  Now, the distanceModulus is automatically added to any
galaxy magnitudes that are calculated, meaning that it will be caught by
arbitrary passbands
add distanceModulus to PhotometryGalaxies calculate_component_magnitudes
* removed old example cosmology mixin
* restored tearDown
* renamed distanceModulus cosmologicalDistanceModulus
* updated cosmology docstrings
* add galid to cosmological test catalog
* added test to make sure that cosmological distance modulus is properly
added to absolute magnitudes
* instead of testing for the existence of self.cosmology, test for
cosmologicalDistanceModulus as a column


(Scott Daniel) Hi Lynne,

are you going to be able to look at this in the next day or so?  There's an API change that Rahul suggested that I think should be implemented.  I opened it as another ticket.  However, if you don't think you will be able to review this pull request soon, I'll just decline this pull request and implement his changes directly on this ticket.

Thanks

-- Scott


(Lynne Jones) Sorry for being so slow, Scott! I'll get to this this afternoon.


(Lynne Jones) Probably swap != None to 'is not None'.


(Lynne Jones) Looks good, all unit tests pass, and documentation is nice. :)


Hard coding these paths makes it phone specific.

Using a command like: vdc volume list | grep 'sdcard ' and then perhaps falling back to something local (nexus 5 doesn't have sdcard mounted through vold).


hey quick question: given this change, how might one build / promote a release version of a given gem to rubygems.org? thanks in advance :)


(Scott Daniel) Bugfix/CATSIM-179 phoSim test is null
* modified Astrometry.py to reflect new spatialBounds API
* modified testAstrometr.py to reflect new spatialBounds API


(Simon Krughoff) Feature/CATSIM-24 generate camera model for catsim
* renamed old get_skyToFocalPlane get_gnomonicProjection and added commments explaining what it does
* added getSkyToFocalPlane based on haversine function
* added unit test to actually write astrometry test star catalog
* unit test now writes the catalog so that it actually tests these getters
* First try at adding chip coordinates
* added conversion to focal plane coordinates
* added utils
* Improved astrometry unit test and added some docstrings


(Scott Daniel) you are missing a third entry and a closing parens in the zip() statement


(Simon Krughoff) I have no idea how the unit test would have run, but I was positive I ran it.  Anyway I fixed the typo and *did* run it this time.


(Scott Daniel) Feature/CATSIM-159 galsim galaxy images
add switch which would allow multiple chips in findChipName (for use with galsim interface, mainly)


(Andrew Connolly) What is the purpose of this flag


How about changing this to filter(None, filters.split(',')) instead?


(Scott Daniel) It's a bit of a hack.  The getter for chipName does not know how to deal with objects that land on multiple chips (which is possible in our test camera object; especially when you consider the difference between science, guider, focus, and wavefront chips).  Rather than force the GalSim interface to worry about the type of each chip, I just added this flag so that the GalSim interface would go ahead and draw objects on every chip they hit.


(Scott Daniel) Bugfix/CATSIM-108 remove trim
changed all use of the word 'trim' to 'phoSim' when referring to phoSim input files (and the variables associated with them).

in conjunction with Bugfix/CATSIM-108-remove-trim in sims_catUtils

* removed 'trim' from Astrometry.py
* removed 'trim' from testAstrometry.py


(Scott Daniel) Bugfix/CATSIM-211 rotskypos in radians
* changed handling of rotSkyPos to reflect new API; unit test fails
* fixed findChipName unit test
* added warning to docstring of findChipName
* clarified error statement


(Scott Daniel) Bugfix/CATSIM-103 update obsMetaData
I'm re-issuing this because I went a little hog-wild in Astrometry.py, adding conditional statements which can raise exceptions, which may have been beyond the scope of this ticket.

* stop skipping unit tests (for the moment)
* changed import statement for Site in testAstrometry.py to reflect that Site has been moved to sims_catalogs_generation
* changed testPassingOfSite to reflect that Site variable is now associated with observationMetaData, rather than the InstanceCatalog
* testAstrometry.py passes after associating Site variable with observationMetaData
* changed Astrometry.py to reflect the new member variables in ObservationMetaData and InstanceCatalog
* adapted Astrometry to the new storage of opsim data in observationMetaData
* changed Astrometry.py to reflect that the observationMetaData variables are associated with catalogs through methods rather than mirror variables
* fixed testAstrometry.py to reflect renaming of obs_metadata.metadata to obs_metadata.phoSimMetadata
* Astrometry.py has been brought in line with changes to ObservationMetaData and InstanceCatalog
* added a bunch of exception raising checks in Astrometry.py


(Simon Krughoff) I still don't understand why these need to be methods.  Can you explain?


(Simon Krughoff) commented out code.


(Scott Daniel) they are now just member variables


(Scott Daniel) Feature/CATSIM-132 split focal plane getters
I have split up the focal-plane related getters.  You can now call calculatePupilCoordinates, calculateFocalPlaneCoordinates, calculatePixelCoordinates, findChipName, and calculateGnomonicProjection independently of writing out a full catalog.  You still need to be in a catalog class, since most of these methods require an observationMetaData object to tell them where the telescope is pointing.

I updated the unit test so that it verifies that these new methods give the same results as the getters (which just call the new methods, so, if this test fails, something awful is happening).

Side note: I also eliminated all of the trailing whitespace from the end of lines in Astrometry.py and testAstrometry.py.  I apologize for all of the meaningless changes Stash will show you on account of this.  The hope is that this will prevent meaningless changes from showing up in Stash in the future.

* changed the name of the database used by testAstrometry.py to be less generic
(so that parallel unittests won't delete each other's databases)
* removed unused d_haversine method
* implemented unit test for testing individual class methods for calculating
focal plane coordinates
so far only works on pupil coordinates
* unit test runs on all focal plane quantities except
x_focal_nominal and y_focal_nominal
* unittest now tests all focal plan class methods
* calculatePixelCoordinates can now accept either
pupil coordinates or ra, dec as arguments (not both)
* calculateFocalPlaneCoordinates now accepts either pupil coordinates
or ra, dec (not both)
* removed trailing whitespace
* findChipName now accepts either pupil coordinates or ra, dec
* added unit test to make sure that focal plane methods raise exceptions
whenever coordinates are improperly specified


(Peter Yoachim) Maybe have an:
__init__(self, camera=None):
   self.camera = camera

or even just set the default to the LSST camera since no one will want to run this without specifying a camera.


(Scott Daniel) Because this is ultimately still a mixin, we don't want to give it an __init__() method for fear of overwriting the __init__() for InstanceCatalog (or whatever the base class ends up being).

As for setting a default: I'm not sure what the state of CameraGeom is.  There may not be an LSST camera class we can default to, yet.

All of the methods in this mixin raise an exception if you have not set self.camera, so the code should be safe.


I'll be watchful of properly formatting defaults in the future. Those brackets are really bewildering, though!

Anyway, changes made. PR is on its way.


(Peter Yoachim) ah, got it.


(Peter Yoachim) The test is dying here for me with an:
AttributeError: 'testCatalog' object has no attribute 'rotSkyPos'


(Scott Daniel) That's an API issue that got changes a few merges ago.  Try pulling down the most recent 'master' on sims_catalogs_measures and sims_catalogs_generation.  That ought to fix the problem.


Since this is a library, we don't want to enforce behavior on the user. If the user doesn't want a stack trace to be printed on keyboardinterrupt, then they can easily catch it in their main as well; I'd rather the exception was bubbled up to the library consumer.


(Peter Yoachim) OK, after much profanity thrown at eups (seriously, you can have two packages simultaneously marked current?!?!), I got most of the tests to pass (looks like I need to upgrade my afw as well, so I'll trust you that it works).


(Scott Daniel) Feature/CATSIM-102 create ebv utils that can be accessed
made equatorialToGalactic and galacticToEquatorial static methods; made getter for galactic coordinates depend on equatorialToGalactic


(Lynne Jones) This looks good and will be useful.


(Scott Daniel) Feature/CATSIM-99 refactor DBObject
refactoring DBObject so that arbitrary SQL queries are possible.  The bulk of the description of this pull request will be in sims_catalogs_generation


(Scott Daniel) Feature/CATSIM-130 update bounds API
see sims_catalogs_generation for a description of this pull request


(Yusra AlSayyad) line up with open paren


(Yusra AlSayyad) Looks good.


(Scott Daniel) Feature/CATSIM-151 split astrometry and photometry
This pull request is the first in a series that will alter the general philosophy behind the Photometry and Astrometry mixins.  When they were written, the mixins were designed to operate solely within the context of an InstanceCatalog.  Every method that did anything interesting was a getter for a database column.  We are now moving to a model where there are, whenever possible, two methods: one method that does the work, but which can be called from outside an InstanceCatalog if so desired, and a getter method that calls the first method.

This pull request touches
sims_catalogs_measures
sims_catUtils
sims_coordUtils
sims_photUtils

* calculateGnomonic projection accepts obs_metadata and epoch as arguments
* fixed calls to applyMeanApparentPlace and applyObservedPlace in
calculateGnomonicProjection
* unit tests for calculateGnomonicProjection
* cleaned up whitespace
* refractionCoefficients takes site as a kwarg
* added unit test for new refractionCoefficients API
* refined docstrings from calculateGnomonicProfjection


(Scott Daniel) Bugfix/CATSIM-189 fix parallax units consistently
* Astrometry.py now expects parallax to be passed in radians;
converts to arcsec before passing to PAL
* added unit test to test that we are handling parallax properly
* cleaned up white space
* documentation


Ok thanks


(Scott Daniel) Feature/SIM-943 move utility functions
* changed import statements to point to sims_utils
* changed imports to reflect that Site.py is in sims_utils


(Scott Daniel) Bugfix/CATSIM-144 cleanup astrometry
I opened this issue when I was trying to write a unit test for generating phoSim catalogs and noticed that our inputs to the palpy routines in the astrometry mixins were not entirely consistent (to wit, we passed MJD for a date when palpy was expecting a julian epoch in years).

I have gone back over Astrometry.py and made sure that all of the arguments we are passing to palpy routines are in the correct units.  I flagged places where palpy expects a date in a specific time system (UTC, UT1, TDB, etc.) and we do not know what time system we are using (currently, all of our dates come from OpSim and are not well-specified).  These will be addressed in a future issue.

This work required some changes to testAstrometry.py.  testAstrometry.py ran slalib routines with nonsensical inputs and then ran the equivalent palpy routines with the same inputs and verified that the outputs were the same.  Now that we have changed the units of some of our calls to palpy, some of the unit tests had to be mangled so that everything remained artificially consistent with that first slalib run.

* unskipped all tests.  Now only testExceptions is failing
* only run testExceptions
* testExceptions passes
had to pick a different test ra and dec for findChipName
* astrometry tests pass again
* cleaned up documentation
* changed calculatePupilCoordinates so that it now calls the haversine
formulat from observationMetadataUtils.py in sims_catalogs_generation
* changed API on applyProperMotion to remove the pm_dec/cos(dec)
* added note that we are going to want to clarify the units of proper
motion in the database
* clarified comments in testAstrometry.py
* eliminated redundant metadata declarations in testAstrometry.py


(Simon Krughoff) Feature/CATSIM-33 database unit tests
Modified test to avoid using UW servers


(Scott Daniel) Feature/SIM-90 use palpy
changes in support of the companion branch on sims_utils

updated getRotTelPos usage


(Scott Daniel) Feature/CATSIM-147 generalize findchipname
This branch makes findChipName (the method which takes a list of RA and Decs and indicates what chips detect the object) callable from outside of the InstanceCatalog framework.  It also makes the methods on which findChipName depends callable from outside of InstanceCatalogs and expands the astrometry unit test to test both the in-InstanceCatalog and out-of-InstanceCatalog APIs for these methods.

Note: in order to make this work (and keep the unit test sensible), I changed findChipName so that it only finds the names of science detectors that see the specified objects.  The test camera provided by AFW contains science detectors and guider chips.  Sometimes, objects fall on both a science detector and a guider chip.  I have altered findChipName to ignore the guider chip in these cases.

An example script showing how to use this code with a simulation of the LSST camera is included in the branch of the same name on sims_catUtils.

You should let me know if you want a different API for MAF.

* changed all of the API exceptions in Astrometry.py to RuntimeError
so that the unit test could catch them
* added the new API to the exceptions tests
* fixed the way applyMeanObservedPlace handled an absent obs_metadata
* added test for exceptions in basic Astrometry methods
* added a layer to the cameraCoords exceptions test to make sure that
the methods ran when given all the appropriate arguments
* fixed whitespace
* moved test for obs_metadata.mjd outside of test for obs_metadata
in correctCoordinates
* unit test tests that exceptions are thrown when obs_metadata
does not carry complete information
* set wrong attribute to None when testing correctCoordinates
* added comment to testAstrometry.py


(Lynne Jones) I have what I think is a minor comment and probably relates to me not knowing the 'bigger' picture for this code in Astrometry.py or what @compound does.
I was looking at the code and trying to understand what things I would expect to use (for MAF, for example) and what things I would not expect to use (because they're tied to the catalogs) and it was pretty clear except for 'AstrometryBase'. There are clearly a lot of methods in there I could use and would be useful, but then there are a few methods which are decorated with @compound that seem like they are tied to the catalogs (since there is a 'self.column_byname' call). So is this a class that is useful external to catalogs, with just a few methods that are related to how it might be used with the catalogs? Or am I misunderstanding that? Maybe some additional documentation near the start would clear this up.


(Lynne Jones) Oh. I think I understand a little better now (a quick google showed up the catalogs documentation on confluence which helped). So we should just ignore those get_* methods.


(Scott Daniel) Yes.  The get_* methods are explicitly for use in catalogs.

Now that we have use cases for these mixins outside of catalogs, I think it might be worthwhile to refactor everything into mixins that just do calculations (with an API that MAF could use) and mixins that wrap those mixins into get_ methods for the catalogs.  We've started to do that haphazardly.  I will open a JIRA issue to do that more systematically.


(Simon Krughoff) You do this several times.  Could you factor this out into a method (maybe defined in this class)?  It would cut down on duplication.


(Simon Krughoff) A few general comments.  Are the extra seemingly unrelated commits about Exceptions part of another ticket, or has master moved on since this PR was created?  If the former, it would help to rebase the ticket to reflect the current master before creating the PR.  If this work was done on this ticket, it would be good to break out unrelated changes into separate PRs because it's a little hard to follow what has changed for what reason.

The fact that SCIENCE and GUIDER chips can both contain the same object is a bug in obs_lsstSim.  Can you send me an example pupil position that shows this?


(Scott Daniel) The exceptions change was done on this ticket.  As I was writing the unit test for this PR, I noticed that, when Astrometry.py was given nonsensical inputs, it sometimes raised a RuntimeError and sometimes raised a ValueError.  I figured it would be better for everyone's sanity if the exceptions were more uniform.  Do you want me to break up the PR now (in which case, can you point me to the git commands that make that possible), or was this a note for the future?

I will send you an example of a faulty camera position in the near future.


(Simon Krughoff) This was really just a style suggestion to make reviews easier.  Keep it in mind in the future.  No need to break up this PR.


(Scott Daniel) The pupil coordinate (1.1e-4, -9.8e-5) lands on the science chip R:1,0, S:0,1 and the guider chip R:1,0, S:0,0.  This is using the test camera generated by afw.cameraGeom.testUtils.camTestUtils.  I will email you a script to highlight the error.


(Scott Daniel) This re-factor was trickier than it should have been.  Do you mind just looking at the changes to testAstrometry.py to make sure I didn't do anything foolish?  The changes revolve around the methods isNanOrNone() and compareTestControlAndWrong()


(Simon Krughoff) I'm a little worried about this type check because it returns False for a unicode string (at least in python2).  What if you first test for None and then catch TypeError from numpy.isnan

    if value is None:
        return True
    try:
        if numpy.isnan(float(value)):
            return True
    except TypeError:
        pass
    return False


(Simon Krughoff) Testing for types can be a little tricky.  String like objects don't have an __iter__ method but all list objects I know of do.  Maybe do this instead:

    if '__iter__' in dir(tt):
        for (t, c, w) in zip(tt, cc, ww):


So, just to be clear, there are 3 branches now:
- 3.x.x is for the old version
- master is what you probably want to developer against now
- remove-activation-conf-fc is my current story/feature branch, which will be merged back into master within a few days or so


Heh - I didn't know what using the curly brace meant  - so I had to go and look it up.


(Simon Krughoff) I had a look and made a couple of small suggestions.  Looks good to me.


(Scott Daniel) thanks.  Got it.


(Scott Daniel) Bugfix/CATSIM-142 add all to modules
This branch adds the __all__= [] formalism to the python source code

added __all__ to Astrometry.py


(Scott Daniel) Bugfix/CATSIM-139 unittest variable scope
This pull request combines two issues:

CATSIM-139 in which we use setUp and tearDown to set and unset unittest instance variables so as to avoid collisions between different unittests

and

CATSIM-115 in which we demand that unittests clean up after themselves so that unnecesssary test outputs don't collect in whatever directory is running the unittests

* rearranged setUpClass and tearDownClass in testAstrometry.py
* testWritingOfCatalog cleans up after itself in testAstrometry.py
* updated tearDown in testAstrometry.py
* fixed whitespace in testAstrometry.py


(Scott Daniel) Bugfix/CATSIM-171 fix parallax units
* parallax is stored in the database in milliarcseconds; convert to arcseconds
for passing to pal
* update docstring
* updated docstrings
* added to comments


(Simon Krughoff) It always makes me uneasy to have a method that has optional arguments that cannot be used if some other arguments are used.  One way to deal with this is to make two methods.  You could also keep this and make it private and expose two @partial methods that do the two different things, just calling with different parameters.


(Simon Krughoff) This is not the way to do this.  We should make it clear what we expect in this method and make the database give us that.  I don't want the database to determine the units the code works in.


(Simon Krughoff) I appreciate that the docstring fixes are in separate commits from the code change, but there should probably be only one of them.  Also, the whitespace commit should be separate from any real changes.


Not so bewildering. The template says [module name] in the ToC where you
would fill in the module name. When you copied the ToC in, you replaced
'module name', but didn't delete the brackets.

On Mon, Apr 27, 2015 at 5:08 PM, Pete Soloway <notifications@github.com>
wrote:

> I'll be watchful of properly formatting defaults in the future. Those
> brackets are really bewildering, though!
>
> Anyway, changes made. PR is on its way.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/puppetlabs/puppetlabs-apt/commit/6581d73f318cfa14970bb6b79372bbcdf66062b6#commitcomment-10929770>
> .
>

--
Jean Bond
Technical Documentation
Puppet Labs


(Scott Daniel) I have opened an issue to that effect (CATSIM-189).  This seemed like the easiest quick-fix in a world where Dave wanted to continue his runs ASAP.


No problem! The latest release, [v0.6.5](https://github.com/kahmali/meteor-restivus/compare/v0.6.4...v0.6.5) should fix this issue and also allows you to [configure default headers](https://github.com/kahmali/meteor-restivus#defaultheaders). If you try the latest version and it gives you any trouble, I'd greatly appreciate your feedback in [Issue #32](https://github.com/kahmali/meteor-restivus/issues/32). Or you can always open a new issue if you bump into any different problems.


(Scott Daniel) What is your concern?

I wanted to keep the API consistent, so that you call the same function name regardless of which parameters you are supplying (I like overloading functions in C++; I'm told this is bad practice, but I like doing it).  It looks to me like you are proposing having separate findChipNameRADec and findChipNamePupilCoords methods (though I admit I only just read the documentation on @partial and still haven't quite digested all of it).  That's too complicated for my taste, but that's just my taste.

Whatever we decide, it will have to be implemented for calculatePixelCoordinates and calculateFocalPlaneCoordinates (since they use a similar API), so I will leave it for another issue.


I think you can use CMSEditLink here right?


(Simon Krughoff) O.K.  You can leave it for another issue.  I think you've captured my intent.  @partial is just a way to minimize the amount of code that is duplicated.  I personally find it confusing when the use of some parameters are dependent on the use of others.    One thing we could think about doing is to pass around a coord like object that knows what type it is so the code can do the right thing.


(Simon Krughoff) I think the right place to fix this is in the CatalogDBObject defiinition:
https://stash.lsstcorp.org/projects/SIM/repos/sims_catutils/browse/python/lsst/sims/catUtils/baseCatalogModels/StarModels.py
by defining a mapping between db value and python side value.


(Scott Daniel) findChipName, calculatePixelCoordinates, and calculateFocalPlane coordinates are meant to be accessible for other applications outside of an InstanceCatalog (i.e. MAF wants to be able to use findChipName).  It seems like making the outside applications use a coordinate object would introduce unnecessary complications.


A simple tic tac toe


I thought so. But the action is copy, which means that you copy the group file from the remote resource into the sandbox.


(Scott Daniel) Feature/CATSIM-130 update bounds API
see sims_catalogs_generation for description of this pull request

updated unittests so that they are in line with the new API
for field of view bounds (e.g. circ_bounds and box_bounds)


(Yusra AlSayyad) OK


(Scott Daniel) Bugfix/CATSIM-95 ra and dec keywords in the metadata
removed redundant getter for raTrim and decTrim in example_utils/exampleCatalogDefinitions/trimExamples.py


(Scott Daniel) Bugfix/CATSIM-113
Scott's work on database-agnostic circle/rectangular search regions


(Simon Krughoff) Remove commented out code


(Simon Krughoff) This import is unused


(Scott Daniel) Bugfix/CATSIM-211 rotskypos in radians
updated unit test to reflect new way of handling
rotskypos in ObservationMetaData


It was moved to a different respository.


(Scott Daniel) Bugfix/CATSIM-103 update obsMetaData
see description in identically named pull request in sims_catalogs_generation

* fixed typo in conditional statement
* added member variables to InstanceCatalog class to reflect the new member variables in observationMetaData
* acces observationMetaData variables using methods, rather than mirror variables
* added a line of whitespace to make code more readable
* rasie error if ask a catalog that has now obs_metadata for its Site information
* added method to give access to bandpass in InstanceCatalog
* added unittest to test how InstanceCatalog handles the metadata variables
* tests default values
* had to add mjd() method to InstanceCatalog
* changes to obs_metadata propagate as expected


fyi, typo: "oder"


(Simon Krughoff) I'd be tempted to require an obs_metadata object (or construct a default one), then you don't need all these if/else.

Also, would you rather do:
self.site = self.obs_metadata.site ?


(Simon Krughoff) We can make this caveat a little less scary by making the attributes of the ObservationMetadata immutable by using the @property decorator.  That way we can raise an error if someone tries to set the variable.  I think that works anyway.


(Scott Daniel) I was able to do away with this caveat by using self.obs_metadata = copy.deepcopy(obs_metadata) to create a deep copy of the input ObservationMetaData instantiation


(Scott Daniel) InstanceCatalog now creates a default obs_metadata if one is not provided.


(Scott Daniel) ---
Also, would you rather do:
self.site = self.obs_metadata.site ?
---

yes. done.


(Scott Daniel) Feature/CATSIM-159 galsim galaxy images
small change in support of the same branch in sims_catUtils

added is_null to list of things to be broadcast from InstanceCatalog.py


(Rahul Biswas) Feature/SIM-1036 column arg in InstanceCatalog
this allows you to add column_outputs as a kwarg of the InstanceCatalog constructor

* default column_outputs to None instead of 'all'
* added column_ouputs arg to InstanceCatalog constructor
* added docstring to InstanceCatalog constructor
* renamed test catalog class in testInstanceCatalog.py
* fixed typo
* added unit test
* added value-based unit test


(Simon Krughoff) Feature/CATSIM-33 database unit tests
* directory structure was mixed up; fixed that
* moved files into new directory structure
* removed Bbox.py
* added decorators
* Added unit test and moved decorators.py from coordUtils
* Moved examples to photUtils and catUtils
* fixed file map and tests.  removed examples and put them in catUtils


So, turns out, returning a `list` in place of an error is kind of problematic because my functions continue to run because they have no idea an error is being thrown. Even in your code in Line 38 and 41 you have to check to see if you get a list back, which isn't a solid way to determine if you got an error. To know if an error is thrown I'd have to check item returned to see if it has a `Rwarning` in it. I'd propose something more like this:

```R
optimalClusters <- function(cols, file_path, n_clust=15,
                            types = c("numeric", "integer"), ...) {
# Your tryCatch should be around your function calls,
# which themselves should use stop("custom error message")
# to surface errors that you can predict.
tryCatch(
    {
        data <- getData(file_path, ...)
    },
    error = function(e) {
        stop(paste('Unable to get data file:', as.character(e))
    }
)
data <- subset(data, select=cols)
# In your checkData function, if the datatypes don't match
# raise an error with stop("Data types do not match"), which
# you will catch here:
tryCatch(
    {
        check <- checkData(data, types)
    },
    error = function(e) {
        stop(paste('Checking of data types failed:', as.character(e))
    }
)
medoids <- pamk(data)
if (medoids$nc > n_clust) {
    stop("The optimal number of clusters is greater than the user-specified max clusters")
}
medoids$nc
```


(Lynne Jones) Imports and dependencies look clean.


(Lynne Jones) Should the weakLensing stuff from here be moved to catUtils? Is it just scientific evaluation/use of the catalogs & instance class? I note that the instance class methods don't use anything from weakLensing, so the whole directory could be moved easily. {and then the instance catalog stuff could go into catalogs_generation more easily and remove the problem of importing catalogs_measures into DBObject?)


(Simon Krughoff) You're right that the weakLensing stuff could move.  I'm going to ask Debbie to do that once the reorg is merged since she may have a better idea if it's useful still.


(Scott Daniel) Feature/SIM-943 move utility functions
* fixed import statements to point to sims_utils
* changed import statements to reflect that Site.py
moved to sims_utils


TeamCity MonoGame :: Develop (Win) [Build 3.4.0.363](http://teamcity.monogame.net/viewLog.html?buildId=6412&buildTypeId=MonoGame_DevelopWin&guest=1) outcome was **SUCCESS**
Summary: Tests passed: 484, ignored: 6 Build time: 00:15:34


(Scott Daniel) Bugfix/CATSIM-188 arcseconds
Some of the classes in baseCatalogModels/GalaxyModels.py incorrectly converted from arcseconds to radians when reading data from the database.  This pull request fixes that bug.  It also modifies the classes in exampleCatalogDefinitions/ to correctly convert back to arcseconds when writing to catalogs.

I also noticed that the class ExampleGalaxyObj in GalaxyModels.py pointed to a table that no longer exists on fatboy, so I deleted it.

* removed ExampleGalaxObj class from GalaxyModels.py.  The table that it
pointed to does not exist on fatboy
* modified GalaxyModels.py to correctly convert arcseconds to radians when
reading from the database
* cleaned up whitespace
* updated phoSimCatalogExamples.py to reflect new arcseconds-vs-radians API
* cleaned up whitespace
* cleaned up whitespace
* cleaned up whitespace
* updated refCatalogExamples.py to reflect new arcseconds-vs-radians API
* fixed bad import statement


(Yusra AlSayyad) Received. Will look later today.


(Scott Daniel) Bugfix/CATSIM-179 phoSim test is null
This is the pull request that I just declined and am re-issuing after modifying the package to be consistent with ObservationMetaData and SpatialBounds storing angles in radians

* also check that control lines are all in test lines
* updated phoSimControlCatalog.txt to reflect the change in astrometry parallax
units
* updated GalaxyModels.py and StarModels.py so that the stored
searches reflected new spatialBounds API
* updates SsmModels.py to reflect new spatialBounds API
* updated testSpatialBounds.py
* eliminated haversine function defined in testSpatialContraints.py
Moved to reliance on haversine function defined in sims_catlogs_generation
observationMetadataUtils.py
* cleaned up whitespace


(Scott Daniel) Hi Simon,  What are the odds that you will be able to review this (and the other related pull requests) in the next day or so?  It provides some functionality that would be nice to have while writing unit tests for the galSim interface.


(Simon Krughoff) Can't you use the deg versions here as you do for the box bounds?


(Simon Krughoff) Same here.  Can you use raDec, decDeg, etc.?


hue


Thank you for the fix


(Simon Krughoff) Same question as with the other two model files.


(Scott Daniel) indeed.  Good point.


(Simon Krughoff) Feature/CATSIM-24 generate camera model for catsim
* added optional setup of obs_lsstSim
* Added tests that call pixel position getters.  Modifies some test metadata to make runtimes reasonable.


(Scott Daniel) The unit test complains about not having a module obsCatalogExamples.  Did you forget to 'git add' something?


(Scott Daniel) disregard this comment.  For some reason, my git did not pull down all of your changes.  I need to fix some things on my system, then I will re-run the test


(Scott Daniel) re-regard this comment.  I re-installed everything and the unit test still complains about obsCatalogExamples.  It did not complain when I ran the unit test in master


(Scott Daniel) you construct the circ_bounds dict differently than you do at lines 37 and 59.  Is there a reason?


Why the &&'s on the ends of the lines?


(Simon Krughoff) Yes.  The reason is that the stars in the dwarf galaxies database are centered at a that location.  To get a reasonable runtime I had to point the observation at that location and make the radius fairly small.


(Simon Krughoff) O.K.  Sorry about this.  I added a file to the repo.


fyi, typo: "or who's"


(Scott Daniel) testAllObjects.py still fails, but only some of the time.  When it fails, I get this traceback

Traceback (most recent call last):
  File "testAllObjects.py", line 24, in testObjects
    result = dbobj.query_columns(obs_metadata=obs_metadata)
  File "/Users/noldor/physics/lsstS2014/Development/sims_catUtils/python/lsst/sims/catUtils/baseCatalogModels/SsmModels.py", line 113, in query_columns
    return ChunkIterator(self, query, chunk_size)
  File "/Users/noldor/physics/lsstS2014/Development/sims_catalogs_generation/python/lsst/sims/catalogs/generation/db/dbConnection.py", line 26, in __init__
    self.exec_query = dbobj.session.execute(query)
  File "/Users/noldor/physics/lsstS2014/DarwinX86/anaconda/1.8.0/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 149, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/Users/noldor/physics/lsstS2014/DarwinX86/anaconda/1.8.0/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 933, in execute
    return self._connection_for_bind(bind, close_with_result=True).execute(
  File "/Users/noldor/physics/lsstS2014/DarwinX86/anaconda/1.8.0/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 803, in _connection_for_bind
    return self.transaction._connection_for_bind(engine)
  File "/Users/noldor/physics/lsstS2014/DarwinX86/anaconda/1.8.0/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 299, in _connection_for_bind
    conn = bind.contextual_connect()
  File "/Users/noldor/physics/lsstS2014/DarwinX86/anaconda/1.8.0/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
    self.pool.connect(),
  File "/Users/noldor/physics/lsstS2014/DarwinX86/anaconda/1.8.0/lib/python2.7/site-packages/sqlalchemy/pool.py", line 272, in connect
    return _ConnectionFairy(self).checkout()
  File "/Users/noldor/physics/lsstS2014/DarwinX86/anaconda/1.8.0/lib/python2.7/site-packages/sqlalchemy/pool.py", line 431, in __init__
    rec = self._connection_record = pool._do_get()
  File "/Users/noldor/physics/lsstS2014/DarwinX86/anaconda/1.8.0/lib/python2.7/site-packages/sqlalchemy/pool.py", line 788, in _do_get
    con = self._create_connection()
  File "/Users/noldor/physics/lsstS2014/DarwinX86/anaconda/1.8.0/lib/python2.7/site-packages/sqlalchemy/pool.py", line 225, in _create_connection
    return _ConnectionRecord(self)
  File "/Users/noldor/physics/lsstS2014/DarwinX86/anaconda/1.8.0/lib/python2.7/site-packages/sqlalchemy/pool.py", line 318, in __init__
    self.connection = self.__connect()
  File "/Users/noldor/physics/lsstS2014/DarwinX86/anaconda/1.8.0/lib/python2.7/ite-packages/sqlalchemy/pool.py", line 379, in __connect
    connection = self.__pool._creator()
  File "/Users/noldor/physics/lsstS2014/DarwinX86/anaconda/1.8.0/lib/python2.7/ite-packages/sqlalchemy/engine/strategies.py", line 80, in connect
    return dialect.connect(*cargs, **cparams)
  File "/Users/noldor/physics/lsstS2014/DarwinX86/anaconda/1.8.0/lib/python2.7ite-packages/sqlalchemy/engine/default.py", line 283, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "pymssql.pyx", line 577, in pymssql.connect (pymssql.c:9000)
    raise OperationalError(e.args[0])
OperationalError: (OperationalError) (20009, 'Net-Lib error during Operation n in progress') None None

It has failed when both in this branch and in master, so I think it is a network connectivity issue (I have tried using both on-campus wireless and wired connections; sometime it succeeds; sometimes it fails).  I'm not sure how we should proceed.  If only humans ever run the unit tests, we can just tell them that sometimes it will fail and that is a network issue.  However, if the automatic stack builds are going to depend on passing the unit tests, that might not be an option.


(Simon Krughoff) Yusra figured out the problem.  The server was overloaded with someone doing a brute force attack on the MS SQL Server.  We are addressing it at the server side.


(Scott Daniel) Feature/CATSIM-159 galsim galaxy images
I have implemented a formalism that directly generates FITS images from the stack.

Note: code in this branch will not run unless you set up the feature/CATSIM-159-galsim-galaxy images branches in sims_catalogs_generation and sims_coordUtils.

The new API is as follows:

Create a GalSim InstanceCatalog based on the GalSimBase class defined in galSimInterface/galSimCatalogs.py.  Write this catalog as you would any other InstanceCatalog (by calling myCatalog.write_catalog('myCatalogName.txt').  As the catalog is written, images are drawn and stored in a way that GalSim understands using the class defined in galSimInterface/galSimInterpreter.py.  After the catalog has been written, call myCatalog.write_images() to actually write out the FITS images to disk.

There is functionality for convolving sources with a PSF.

There is also rudimentary functionality for adding noise to the image.  However, this is not rigorously implemented or tested.  The current interface between CatSim and OpSim is far out of date.  It is not straightforward to read things like sky brightness from OpSim into CatSim.  During our next sprint, I will update the CatSim-OpSim interface.  Once that is done, I will implement a more rigorous noise model.  That being said, the noise model is designed so that users can write their own wrappers for GalSim noise classes, so any given user ought to be able to implement a more realistic noise model without a lot of work (it just won't be able to talk to OpSim yet).

The unit tests generate noiseless FITS images and then read them back in with AFW, calculate the number of counts in the images, and compare that to the expected number of counts calculated from the objects' SEDs using photUtils.

There are example scripts in sims_catUtils/examples/galSimExampleScripts/ demonstrating how to use the existing functionality.

I will work with Simon over the next few days to make GalSim itself eups distib installable so that we can actually merge this ticket to master without breaking buildbot.

* removed **kwargs from drawObject formalism
* documented galSimInterpreter.py
* refere to detectors rather than chips
* fixed unit test so we can account for objects on multiple detectors
also moved objects away from detector edges so we don't have to
* commented unit test
* do not test very dim images
* documented example scripts
* added example noise implementation
* re-factored PSF and noise to be in their own source code file
fixed bug in noise examples
* moved GalSim example scripts into their own directory


(Andrew Connolly) typo on examle


(Andrew Connolly) stars not 2D sersics?


(Scott Daniel) No.  Stars are instances of one of GalSim's PSF classes.  Should that not be so?


(Andrew Connolly) remove whitespace


(Andrew Connolly) Not sure I follow. In the comments it says you are using 2D sersics. If you are going to convolve with a PSF I would assume that you need would pass a point source (delta fn wrt to the pixel scale).


(Scott Daniel) As I understand GalSim: there is a class of objects called PSF.  You can just draw in image that "is" a PSF.  My interpretation of that is that it is a point source with a given flux convolved with a PSF.

There is no class or function GalSim for producing a delta function source.  They just have Sersic profiles and PSFs.  You can convolve them together.  I thought Sersic profiles were only for describing extended objects.


Take away the `deriving Eq` from here.  Instead, just have `deriving (Show, Read)` and then add an explicit `instance (Graph gr, Eq a, Eq b) => Eq (OrdGr gr a b) where ...`.

This instance can either:
1. Use the `equal` function (will work, but not efficiently)
2. Use the `Ord` instance (yes, this is OK!) with `g1 == g2 = compare g1 g2 == EQ`; this will require changing the constraints to `Ord a, Ord b`.
3. (not recommended) duplicate the compare definition below but change it to use `==`.

I think in this case it's probably best to follow @cmaeder's advice and take 2.


TeamCity Architecture Team :: Java Platform :: otj-logging :: build-config [Build 17](http://teamcity.otenv.com/viewLog.html?buildId=1277617&buildTypeId=ArchitectureTeam_JavaPlatform_OtjLogging_BuildConfig) is now running


TeamCity Architecture Team :: Java Platform :: otj-logging :: build-config [Build 18](http://teamcity.otenv.com/viewLog.html?buildId=1277618&buildTypeId=ArchitectureTeam_JavaPlatform_OtjLogging_BuildConfig) is now running


(Andrew Connolly) to use


(Andrew Connolly) Is it worth noting where this and the noise are defined


TeamCity Architecture Team :: Java Platform :: otj-logging :: build-config [Build 18](http://teamcity.otenv.com/viewLog.html?buildId=1277618&buildTypeId=ArchitectureTeam_JavaPlatform_OtjLogging_BuildConfig) outcome was **FAILURE**
Summary: Exit code 1 (new); exit code 2 (new) Build time: 0:0:0


Thanks for this Jayson. I intend to get this merged and deployed today. One small note, when doing stylistic changes like changing whitespace, hash syntax, etc. please commit these changes in a separate commit. As it stands it's not immediately obvious what are style changes and what are actual test changes when they're all mixed together.


(Andrew Connolly) In general I don't think it is a good idea to import * (what does the LSST style guide say @krughoff). Can you isolate the routines or import as


You could use single quotes instead of double quotes for the inner quotes. The $opt_trace_name will still be expanded (because it's the outer quotes that determine the expansion of variables).


(Andrew Connolly) If you assume a Gaussian and a given brightness you could calculate the radius away from the edge of a detector based on surface brightness) For bright sources it will be over an arcmin


(Andrew Connolly) I think you need to pass brightness to this function


(Andrew Connolly) Where do we write the correct metadata (header keywords to pass the data into the DM pipelines)


Once you've added an explicit `Eq` instance for `OrdGr` that no longer requires on there being an `Eq` instance for `gr`, you can remove that constraint from here.

Once you do that, the `FlexibleContexts` extension won't be needed.


(Scott Daniel) We do not.  I have not yet looked into what will be necessary to run these FITS images through the DM piplelines.  That and correctly implementing noise are the next step of this project (where "this project" is making GalSim talk to the stack).


(Scott Daniel) I actually think I can do this more exactly using GalSim's image class.  For each object, I can just draw a test image, and then compare all of the non-zero pixels in the image to the bounds of the detectors and thus determine all of the detectors on which the object casts light.

This will take some re-factoring of the drawObject and findAllChips methods, so I will make it an issue in the February 25 sprint (since the February 11 sprint is already planned).

For now, I will just increase the conservative radius so that any object with 2 arcminutes of the detector is drawn on that detector.


hey man.


There's debugging and there's debugging. Let's cut this ;-)


Should this be bash instead of sh?


 Thank you for your quick reply.

I test it with recent [nfdump-1.6.13](http://sourceforge.net/projects/nfdump/), building configure option '--enable-nsel'.
ulogd.conf is very simple: (a scribbled memo for me)

    [global]
    logfile="/var/log/ulogd.log"
    loglevel=1

    plugin="/usr/local/lib/ulogd/ulogd_inpflow_NFCT.so"
    plugin="/usr/local/lib/ulogd/ulogd_filter_TIMECONV.so"
    plugin="/usr/local/lib/ulogd/ulogd_filter_PACKICMP.so"
    plugin="/usr/local/lib/ulogd/ulogd_output_NFLOW9.so"

    stack=ct1:NFCT,packicmp:PACKICMP,timeconv:TIMECONV,nflow:NFLOW9

    [ct1]
    hash_enable=1

    [nflow]
    dest="udp://127.0.0.1:9995"

    [timeconv]

on terminal

    # mkdir nfdata
    # nfcapd -D -l nfdata
    # ulogd -v
    (for a while... and Ctrl-C)
    # killall nfcapd
    # nfdump -R nfdata


TeamCity MonoGame :: Develop (Win) [Build ???](http://teamcity.monogame.net/viewLog.html?buildId=6415&buildTypeId=MonoGame_DevelopWin&guest=1) outcome was **FAILURE**
Summary: Snapshot dependency failed: MonoGame :: Develop (Mac) Build time: 00:00:00


So I tried your suggestion, but it doesn't work because `ByteBufOutputStream` doesn't do short writes; if you have a file region that has more data than the buffer size, it will try to write all available bytes and throw an exception because the backing buffer is too small.

At this point, I think `ByteArrayWritableChannel` is the way to go, so I'll submit updates restoring that code.


Can you give us the rationale for this commit? What was being fixed?


This overall is not that cheap to do. And most of the time it will be just for a no-op. Can we wrap it in a check whether the bottomInset will actually be changed on either set of insets?


Simon bato!
![cool-rose-hd-wallpaper](https://cloud.githubusercontent.com/assets/8831497/7360667/7474d70a-ed02-11e4-87c8-48764dcfaa1a.jpg)


![screen shot 2015-04-17 at 10 41 06 am](https://cloud.githubusercontent.com/assets/10620436/7360669/a2b59708-ed1b-11e4-8597-55b2859da403.png)


nice


develop (lefticus) - Win64-Windows-8.1-VisualStudio-12: OK (345 of 346 tests passed)

![Build Badge](http://img.shields.io/badge/build%20status-passing-green.svg) ![Test Badge](http://img.shields.io/badge/tests%20passed-99.71%25-yellow.svg)


Should this be indicating what is not a symbol?  Or is that just an MRI quirk?


It was already there! Removing it. ;)


I added this and tested it; it works. Thanks!


@sirmordred Test with the latest device changes I pushed.


smaller_make_shared (lefticus) - i386-Windows-8.1-VisualStudio-14: OK (346 of 346 tests passed)

![Build Badge](http://img.shields.io/badge/build%20status-passing-green.svg) ![Test Badge](http://img.shields.io/badge/tests%20passed-100.0%25-green.svg)


Hi all,

please have a look to the change of the build.gradle file to fix the issue related to invalid links to files. I'am not sure if it is Ok to copy the resource files to the html5 directory?


I have a suspicion that this horrible thing (ScreenUpdateSetting) is a hack to fix some underlying OpenGL defect(s) which the original Rice Video authors were never able to figure out.  I don't think any other video plugin has a setting like this, and the fact that it behaves differently on different OSes suggests to me that hiding a deeper bug.


1. s/subtraction of P/subtraction of Ord(P)/.
2. I don't understand the references to Lpoly[0] and Lpoly[1] in the comments. Are these supposed to be mentioning Lord[0]...Lord[3] instead?


Could you do me a quick favour and help me understand this one?  Why would a cast be necessary here?

```my_it->first``` is ```int64_t``` and since any ```int64_t``` value can fit in ```int128_t``` I'm not sure I understand what the difference is between:
```
int128_t owns = my_it->first;
int128_t owns = int128_t(my_it->first);
```

Thanks dude :)


lol too funny o well shit happens :-D


@javacruft, would you mind checking that this looks correct to you?


TeamCity FubuMVC Regex Url Policy :: Deploy [Build 1.0.21.0](http://build.mikeobrien.net/viewLog.html?buildId=2302&buildTypeId=bt18&guest=1) outcome was **SUCCESS**
Summary: Tests passed: 57 Build time: 00:01:04


@nirvdrum They usually include more detail. I think the error message needs to be updated to (ruby pseudo-code)
```ruby
"#{arg.inspect} is not a symbol"
```
Some MRI Examples
```
private_class_method Object.new
#<Object:0x007f973195fc48> is not a symbol

private_class_method ["test"]
["test"] is not a symbol (TypeError)


Same here. IIRC, this should be "Subtract Ord(P)", or better "Branch-less conditional subtraction of Ord(P)".


`false` means...?  I don't suppose I could convince you to change this to an `enum class`?


smaller_make_shared (lefticus) - Win64-Windows-8.1-VisualStudio-14: OK (346 of 346 tests passed)

![Build Badge](http://img.shields.io/badge/build%20status-passing-green.svg) ![Test Badge](http://img.shields.io/badge/tests%20passed-100.0%25-green.svg)


could just be `auto`


ditto.


Ah, right - that needs to be `TRANSFER` then, thanks!


erm for allied units pos1 will be uninit


Not in this branch.


@rpaquay: `!0`, `!''`, and some others will all evaluate to `true`, giving you hard-to-debug false positives in cases like this. So `!something` should be used with caution and full understanding of the implications. To test if a key is present in an object, `if (key in obj)` is the safest and most readable, I think.


Errm, floating point equality check. Prefer ABS(insets.bottom - bottomInset) > FLOAT_MIN or similar.


# Segunda Parte:

## Lista de atividades:
1. [X] Adicionar um diretor a uma filme já existente.
2. [ ] Encontrar os filmes do mesmo diretor do filme corrente.
3. [ ] Lidar com a busca por filmes do mesmo diretor, mas o filme corrente não possui diretor.

### Atividade Um
1. Adicionar os Cenários. Para isso copie os cenários linkados no site Pastebin. O Link esta na descrição da atividade.
2. Adicionar um campo na página de edição para permitir a entrada de um diretor pelo usuário.
3. Adicionar o arquivo **movie_steps.rb**.
4. Adicionar um passo para colocar os filmes no banco.
  Utilizando o método `hashes` do movie_table e `each`para percorrer a tabela. Então, adiciona-se pelo `create( )` os filmes na tabela.

5. Adicionar um passo para analisar se o diretor corresponde com que está na tabela.
  Usando a captura dos elementos na regex `/the director of "(.*)" should be "(.*)` procura-se o título do filme no banco e compara o diretor que está vinculado ao filme com o capturado na regex, que é o que deveria ser.

**Observação**: Os dois últimos cenários estão comentados aqui, pois não estão sendo implementados agora.


Merge branch 'master' of github.com:devsnd/cherrymusic


Aarrgh


You can use `sys.env.get("publishTo")` to get an `Option[String]` here, too


Just noticed the nifty Start/Stop button - good work :+1:


Regarding elo gain, don't get over-excited yet. I expect about 0 elo with this version. Perhaps a 1-2 elo gain with a bit of luck, still a bit below what we can reliably detect with SPRT.
That's still encouraging, considering I started from scratch, and crafted that PSQT in just a few days with only my quad for tuning and testing.


Going through the cases

1. If previousData is null, then currentData has nothing in it.  We try to put data into currentData and if that returns a null, that means we successfully put something and the previous value was null, so all is good and break out of the loop.  If our put failed, that means `currentData` was updated, so we loop back around.  If `data` is null, then we definitely could enter an infinite loop.  We should perhaps make sure that `data` is never `null`?
2. If we have a version that is newer, we then go ahead and try to update our state conditioned on it still pointing to what we think it does, if that succeeds, we break the loop.  If that fails, it's because the state of currentData has changed, so we loop back around to get the updated state.
3. Break out of the loop

It's an "optimistic", "live" thread-safe implementation.  If you are worried about it, though, we could essentially do the same thing that used to be done but use synchronization instead?


smaller_make_shared (lefticus) - i386-Windows-8.1-VisualStudio-12: OK (345 of 346 tests passed)

![Build Badge](http://img.shields.io/badge/build%20status-passing-green.svg) ![Test Badge](http://img.shields.io/badge/tests%20passed-99.71%25-yellow.svg)


https://github.com/meteor/mobile-packages/pull/36
On Apr 27, 2015 5:31 PM, "Dan Dascalescu" <notifications@github.com> wrote:

> Yes, how about a PR against the original package?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/cscott/mobile-packages/commit/2ca44285fa4431f87943378ad77d9245cb0ff0b3#commitcomment-10927085>
> .
>


Can you avoid allocating a new array here. You can reuse an array by setting its `length` property, or even create a pool of arrays of different sizes, see `makeArrays` in `utilities.ts`.


Good point. I was in JavaScript mode all day today so didn't think much of the null until after merging :)


bad formatting


Also, I wouldn't add dynamic keys to objects like here. I'd use an explicit catch-all bucket for custom streams, e.g. `NestedStdio.customStreams`. With dynamic keys, you'd see usages like `someNestedStdio.overlay` in the code and wonder where `overlay` has come from. With `someNestedStdio.customStreams.overlay`, you can search for `customStreams.*overlay` and get a much smaller set of candidates.


you sure this improves ? Thanks


I submitted the PR three months ago.


Why not `stdio.overlay`?


That `&` should be `&`. I think.


Ahh, yep! That explains it. Bit embarrassing, though.


@chanwit Just saw you have already worked on this :). I just created a token discovery test https://github.com/docker/swarm/compare/master...snrism:token_integration_test, although the "swam join" step fails in my setup (still debugging). Any reason why this is not merged yet? Thanks.


It always been the case


This change has mistake that update skin src when check with container at line 341.


@deavmi
> The Frefunk logo

Is that spelled differently on purpose? I don't know anything about them, so I wouldn't know.


Что значит человек == жопа!??? =)


Thank you so much.


Great!


И все остальные


Inden mismatch


Indent mismatch


Hey Cala could you also add the "updates" from v1.5.0 please? Only the updates not the whole DB.
Regards


The protocol part looks good.


Missing a closing tag here.


missing a closing tag here


(Scott Daniel) I actually think I can do this more exactly using GalSim's image class.  For each object, I can just draw a test image, and then compare all of the non-zero pixels in the image to the bounds of the detectors and thus determine all of the detectors on which the object casts light.

This will take some re-factoring of the drawObject and findAllChips methods, so I will make it an issue in the February 25 sprint (since the February 11 sprint is already planned).

For now, I will just increase the conservative radius so that any object with 2 arcminutes of the detector is drawn on that detector.


(Andrew Connolly) I'm not sure that the "generate all images and test" is the most effective way as you will still have to generate the image and you will still have to decide which objects you will select to generate that image. So you will still need a heuristic for which objects to use to create the test image


(Scott Daniel) Right now, the way the interface works is this:

For each astronomical object and each detector that the object might fall on, we create an image of the object.  That image is then added to the image associated with the detector (see the end of the drawObject method in galSimInterpreter.py).

This means that, every time our "if I approximate the image as a Gaussian.... it will probably fall on these chips" method gets it wrong (decides that the object is casting light on a chip it actually isn't), we are already drawing an image of the object that is unnecessary.

Drawing one test image of the object for purposes of determining what chips it illuminates means, at most, drawing one wasteful image.  The current system draws several wasteful images.

For reference, I tried writing the code so that it only drew image after each CCD was assembled.  This caused GalSim to throw an error whenever a faint object occurred on a chip with a bright object.  The faint object was ignored due to rounding errors, and GalSim complained.  I could not find a way to circumvent the error.


(Scott Daniel) Feature/CATSIM-78 accept sdss filters in photutils
This is just the sdss getters from the photUtils branch of the same name moved into a file that shows how to write getters for arbitrary bandpasses

* added exampleSDSScatalogs.py
* exampleSDSScatalogs.py runs


(Simon Krughoff) Sorry to do this to you since I originally named these, but can we choose something other than *Recalc?  It wasn't a very good name.  Unless it's overloaded, I think we can just use sdss_[grizy].


(Scott Daniel) Bugfix/CATSIM-96 galaxies have no size
Somehow, our code to generate trim files for phoSim was using improper syntax when naming spatialModels.  This is just an attempt to fix that.

The fix occurs in /lsst/sims/catUtils/exampleCatalogDefinitions/trimExamples.py

I also added a file examples/generatePhosimInput.py which Andy Connolly has been using to generate trim files.

There is a related pull request in sims_catalogs_generation which removes some code which had tried to set phoSim spatialModels in the DBObject class, which we decided was a bad ide.

* spatialModel is now wholly contained within trimExamples.py
* added generatePhosimInput.py
* removed all of the vain references to my own initials from generatPhosimInput.py
* generatePhosimInput.py now runs
* added comment at top of generatePhosimInput.py explaining what the file is


(Simon Krughoff) I'd like to remove mention of "trim" files.  This is CATSIM-108.  I'm not asking that this be done as part of this issue, but maybe updating the docstrings here would help down the line.


(Simon Krughoff) Do we need to be careful of case sensitivity?  I could have sworn that PhoSim wanted all caps.


(Scott Daniel) I checked the phoSim source code.  It will accept all caps or all lower case.  I have run these trim files through PhoSim and they seem to work.


(Scott Daniel) what are we going to call them in place of "trim files"?


(danielsf fixed) Bugfix/CATSIM-108 remove trim
replaced all use of the word 'trim' to 'phoSim' when referring to phoSim input files and the variables associated with them

in conjunction with Bugfix/CATSIM-108-remove-trim in sims_coordUtils

* removed 'trim' form trimExamples.py
* renamed trimExamples.py to phoSimCatalogExamples.py
* removed 'trim' from catalogsExamples.py
* updated __init__.py in exampleCatalogDefinitions/
* removed 'trim' from generatePhosimInput.py


(Scott Daniel) Bugfix/CATSIM-211 rotskypos in radians
updated findChipName example script


(Scott Daniel) Bugfix/CATSIM-103 update obsMetaData
see description in identically named pull request in sims_catalogs_generation

* removed superfluous assignment of mjd from OpSim3_61_DBObject.py
* updated 'trim' file code to reflect renaming of obs_metadata.metadata to obs_metadata.phoSimMetadata
* had to update unit test to reflect the obs_metadata.metadata -> obs_metadata.phoSimMetadata renaming


(Scott Daniel) Feature/SIM-952 realistic galsim psf
Replaced the nonsense PSF classes I had implemented for the GalSim interface with something a little more realistic

* wrote DoubleGaussianPSF class
* removed all mention of ExampleOpticalPSF
* added unit test for GalSim PSF


(Peter Yoachim) seems like you could make it general my having kwargs for fwhm1 and fwhm2, and kwargs to set the weights since those seem like magic numebrs.  But far be it from me to question the SNR document if it's all coming from that.


(Scott Daniel) good point.  I will rename this class SNRdocumentPSF to make it clear that it is not a general double gaussian psf.  I will also add a DoubleGaussianPSF class that is more general.


Curious what happens w/o this patch?


(Scott Daniel) Bugfix/CATSIM-169 catalogModel bugs
There were some typos in the mappings between the database schema and the column names expected by CatSim in the baseCatalogModel catalog classes.  This fixes them.

* fixed mapping of proper motion from database to catalog model classes
* fixed typo in LensGalaxyObj class


(Scott Daniel) Feature/CATSIM-99 refactor DBObject
refactoring DBObject so that arbitrary SQL queries are possible.  The bulk of the description of this pull request will be in sims_catalogs_generation


(Simon Krughoff) This was always a little bit of a hack.  Maybe it would be better to use DBObject here  since we don't need the registry.


(Scott Daniel) the getObservationMetaData method in this class relies on the CatalogDBobject infrastructure.  I will file a separate JIRA issue to disentangle all of this


any update on this?


(Scott Daniel) Bugfix/CATSIM-189 fix parallax units consistently
This branch fixes the parallax units problem in a way that is consistent
with our philosophy.

There are branches in
sims_catalogs_generation
sims_coordUtils
sims_catUtils

added code to convert parallax from milliarcseconds to radians
when reading into a CatalogDBObject class


(Scott Daniel) 'cause I'm a nagging nagger who nags


(Scott Daniel) Feature/CATSIM-151 split astrometry and photometry
This pull request is the first in a series that will alter the general philosophy behind the Photometry and Astrometry mixins.  When they were written, the mixins were designed to operate solely within the context of an InstanceCatalog.  Every method that did anything interesting was a getter for a database column.  We are now moving to a model where there are, whenever possible, two methods: one method that does the work, but which can be called from outside an InstanceCatalog if so desired, and a getter method that calls the first method.

This pull request touches
sims_catalogs_measures
sims_catUtils
sims_coordUtils
sims_photUtils

updated exampleSDSScatalogs.py so that setupPhiArray_dict is no longer
explicitly called (it is called by loadBandPassesFromFiles)


(Scott Daniel) Bugfix/CATSIM-140 opsim as dbobject
OpSim3_61DBObject used to inherit from CatalogDBObject (the class which interfaces catalogs with the database).  This was just a hold-over from when there was only one DBObject class and was probably unnecessary.  After this pull request, it will inherit from DBObject (the class which interfaces with the database generically).

* catalogsExamples.py runs
* removed unnecessary objid from OpSim3_61DBObject
* cleaned up some whitespace
* removed more unnecessary member variables from OpSim3_61DBObject
* removed unnecessary import from OpSim3_61DBObject.py
* cleaned up whitespace
* added documentation to OpSim3_61DBObject.py
also removed an unnecessary kwarg from the getObservationMetaData definition
* added some helpful whitespace
* added some helpful whitespace
* added comment


(Peter Yoachim) Now there are lots of new Opsim runs that will hopefully replace 3.61 as the default baseline survey.  Obviously, updating to the new runs that have slightly changed schemas will be a ticket for another day.


(Yusra AlSayyad) Bugfix/CATSIM-209 queries to allstars return duplicate
This fix ended up being just one line.  I replaced fHtmCoverBinaryAdvanced with fHtmCoverRegion in StarModels. fHtmCoverRegion only returns the outer  htmids (instead of the inner and outer htmids like fHtmCoverBinaryAdvanced). Tests pass and so does your repetitiveStars example.


(Scott Daniel) Feature/SIM-166 update opsim interface
This branch introduces a class (ObservationMetaDataGenerator) that reads an opsim output database and returns a list of ObservationMetaData instantiations that conform to requirements set by the user.

It will require you to setup the feature/SIM-166-update-opsim-interface branches in

sims_catalogs_generation
sims_data (a new repository for holding data in support of unit tests)

* updated documentationin ObservationMetaDataGenerator.py
* cleaned up whitespace in ObservationMetaDataGenerator.py
* use list comprehension for phoSimMetadata
* use list comprehension on obs_output
* make columnMapping a list rather than an OrderedDict
* stop importing OrderedDict in unit test
* actually test that PhoSim headers are correctly made
* add comment
* add test for filter
* formatting


(Scott Daniel) Did we ever figure out what happened to rotTelPos in OpSim?  It occurs to me that PhoSim expects rotTelPos.  I can make adding a default value for rotTelPos a part of "update the OpSim interface to match what PhoSim expects" issue.


(Peter Yoachim) oops, forgot to answer this.  It looks like it disappeared totally from the sqlite file.  Go ahead and file a ticket on Opsim to update the schema and put it back in.  RotSkyPos is in there, but I don't think we want to recalc rotTelPos all over the place, it should just be in the sqlite.


(Scott Daniel) Feature/SIM-136 photometric uncertainties
see description in corresponding sims_photUtils branch for explanation

* fixed bug which meant that testObservationMetaDataGenerator.py was not
actually testing queries done on m5
* updated exampleSDSScatalogs.py to reflect change from
bandPassList to bandpassDict in Photometry mixin
* added unit test for photometric uncertainty getters
* made error message more helpful in
PhoSim unit test
* added class wrapping GalaxyTileObj to the test utils
* added unit test for galaxy photometric uncertainty getters
* fixed bug in unit test
* fixed examples to comport with new API
* documented testGetters.py
* sped up testGetters.py


(Lynne Jones) Hey, I'm glad to see you're using the env variables like this (and not "SDSS_THROUGHPUTS", since I think I'm going to have to get rid of those extra ones).


AFAIR grub2-mkconfig is very careful which variables got exported into generator scripts.
To get that working, you probably need to explicitly export GRUB_CMDLINE_TBOOT in /etc/default/grub (not only set it value). Worth mentioning somewhere.


(Lynne Jones) I wonder if these two "ifs" are really equivalent - could someone have a bandpassDict defined, but no phiArray yet?


(Scott Daniel) Probably.  I am erring on the side of paranoia


(Scott Daniel) Feature/CATSIM-91 simplify phoSim input
I will describe the point of this branch in the related sims_catalogs_measures branch

* made radius of field of view smaller for testing purposes
* PhoSimInput base now relies on the list of cannot_be_null columns to prevent
a phoSim input file from containing objects wit 'None' for the SED file.
Part of this change was altering get_sedFilepath so that it would return
None as an output if the SED file path did not exist (previously, the
code would just raise an exception)
* reverted to old radius in generatePhoSimInput.py


(Scott Daniel) Feature/SIM-943 move utility functions
* changed import statements to point to sims_utils
* removed unnecessary import statements
* updated documentation in tutorial00


(Scott Daniel) Feature/SIM-950 refactor galsim findDetectors
This branch changes how the GalSim interface determines whether or not an astronomical object casts light on a given chip.

Before, a heuristic rule was applied, comparing the size (multiplied by a scale factor) and position of the object with the bounding box of the chip.  Now, a test image is drawn and pixels in that test image with non-zero flux are compared to the bounding box of the chip.  If non-zero pixels fall within the chip, the object is said to illumine the chip.

* do away with need to write catalog before writing images
* only print out u, g, r filters in GalSim unit test
* write_images returns the names of the files written
* added unit test to check that objects are placed on the right chip
* replaced x_pupil and y_pupil with xPupil and yPupil (except
in the case of the catalog columns; that would require touching
sims_coordUtils, which I don't want to do now) for consistency
* changed unit test to account for the probabilistic nature of photon shooting
* updated docstrings in galSimCatalogs.py
* updated docstrings and comments in galSimInterpreter.py
* make sure that some detectors have 'zero' flux
* documented new unit test


(Scott Daniel) I just (March 12) rebased this to master so I could use it for my Tucson demos.  You'll want to do a pull before attempting to review this.


(Bryce Kalmbach) typo here, "tothe"


This commit broke the test suite.


(Bryce Kalmbach) typo, "calss"


(Bryce Kalmbach) typo, "empy"


(Bryce Kalmbach) typo, "positionaAngle"


(Bryce Kalmbach) typo, "Sersci"


(Bryce Kalmbach) Are the extra empty lines necessary?


(Bryce Kalmbach) Is there any way to come up with an actual rule here?


(Bryce Kalmbach) Is the reason that there is a all detector loop and then a viable detector loop the fact that these two operations take an appreciable amount of time or is there something else?


(Bryce Kalmbach) This line and others below exceed 110 columns.


(Bryce Kalmbach) This line and some others below in this test exceed 110 columns.


(Scott Daniel) I don't think so.  Galaxy components half a half light radius, so you can just reject chips that are a certain number of half light radii away from the centroid of the galaxy.  The trick is point sources.  Since there are many forms the point spread function might take, there's no way to code a similar rule using a general "characteristic length" of the point spread function.


(Scott Daniel) Yes.  It's all about speed.  _doesObjectImpingeOnDetector has to loop through all of the non-zero pixels in the test image.  I am just trying to limit the number of times that happens.


(Scott Daniel) Feature/SIM-120 ipython demos
This branch takes some demos that existed as python scripts and makes them iPython notebooks

* replaced commented tutorial03.py with iPython notebook
* replaced commented tutorial04.py with iPython notebook
* proofreading of tutorial00.ipynb
* proofreading tutorial01.ipynb
* proofreading tutorial02.ipynb
* proofreading tutorial03.ipynb
* proofreading tutorial04.ipynb
* added application to the end of tutorial04.ipynb
* brought tutorial04.py in line with tutorial04.ipynb
* cleaned up whitespace


(Rahul Biswas) I am not sure I understand the variable `catalogType`. Are there an allowed list of strings that are sensible? What is the purpose of this variable?


(Rahul Biswas) Are
`TutorialCatalog(myDB, obs_metadata=obs_metadata)` and `myDB.getCatalog('tutorial_catalog', obs_metadata=obs_metadata)`
exactly equivalent (why have two ways to do the exact same thing)? Or are there situations where one may prefer one over the other?


(Rahul Biswas) - It would be good to move this function lower down in the notebook, to where it is being used. It is a little confusing here at this location.
- I am guessing this code is not in sims_utils or something like that. I use almost this code to write SN light curves to a database. It would be useful to be able to use a common code and not duplicate them.


(Scott Daniel) Yes.  They are equivalent.  I'm not sure why that design decision was made.  Maybe there are cases where the user is already manipulating the CatalogDBObject and it is just easier to use getCatalog rather than having to instantiate the InstanceCatalog class by hand.  I could just be making that up, though.


(Scott Daniel) catalog_type is the argument that is passed to CatalogDBObject.getCatalog()


(Rahul Biswas) Yes, I understand that. But what information is that supposed to carry. What are the possible string values aside from 'tutorial_catalog' and what will the differences be?


(Scott Daniel) it is literally just a nickname you give to the catalog class so that getCatlog() can find it.  If you do not assign one, the InstanceCatalog meta class creates one for you by (I think) converting the camel case name of the class into a string that is separated by '_' (so myStellarCatalog becomes my_stellar_catalog)


(Scott Daniel) each class has to have its own unique identifier, or else the meta class raises an exception


(Scott Daniel) This is where the code is used.  You have to create the database before you can connect to it using a CatalogDBObject.

I agree that we need to consolidate our database-writing code.  There are several routines that are floating around for generating cartoon databases to be used with unit tests.  I'm not sure that these should be unified with code to write database for actual CatSim structures (i.e. the supernova catalogs).  It seems that those are different use cases: test databases are designed to mimic fatboy; things like the supernova catalog databases provide new databases of unique results.


(Yusra AlSayyad) Bugfix/SIM-994 galaxyagnobj and galaxybulgeobj need
* Add parameter to GalaxySearch stored procedure:
@componentSubset which takes one of 'ALL', 'AGN', 'BULGE', 'DISK' to limit search
to only galaxies having those components
* Remove unused attribute tableid from GalaxyTileObj subclasses


(Scott Daniel) Bugfix/CATSIM-131 phosim unittest
This branch adds a unit test that generates a PhoSim input file and then verifies that file against the output of the same unit test run under controlled circumstances.  This means that if we ever change the logic in Astrometry.py (which we likely will if we decide to acknowledge the difference between UTC, UT1, and TDB time systems), we will have to regenerate the data file for this unit test.  I believe this is a feature, as it means that this unit test will act as a warning system should we ever accidentally change the way Astrometry.py behaves.  I can be talked out of that position (in which case, this unit test will just verify that PhoSim input file has been created without raising an exception).

* wrote code that creates a PhoSim input file based on a cartoon database
next step is to turn it into a proper unit test
* removed diagnostic print statements and some unhelpful comments
* added a baseline catalog against which to test the output of the
phoSim catalogs
* implemented a unittest that writes a phoSim input catalog and compares
to  previously generated phoSim input catalog that used the same
inputs
* fixed whitespace errors
* remove unneeded galtileid definition
* add comments and docstrings to the unit test
* added comments to the top of testPhoSimCatalogs.py warning that,
if the logic in the other packages changes, the unittest will fail
* assign testSite to ObservationMetaData


(Simon Krughoff) This is unrelated, but should we look into using PAL to do this conversion rather than the methods I use?  It makes things more maintainable to have one way of doing things.


(Simon Krughoff) I must be missing something.  Why aren't you using (or extending) the code that already does this in:
https://stash.lsstcorp.org/projects/SIM/repos/sims_catalogs_generation/browse/python/lsst/sims/catalogs/generation/utils/testUtils.py

I'd much prefer to keep this kind of utility code for tests in some common location so that many tests can use it.


Oops, you are right. But shouldn't we use some *epsilon* instead of `CGFLOAT_MIN`?

For example, having:

    (lldb) po bottomInset
    216
   (lldb) po insets.bottom
    216.00000177001994

`ABS(insets.bottom - bottomInset) > CGFLOAT_MIN` returns true (when it shouldn't).


(Simon Krughoff) I think the preferred way to do this is with eups.product_dir('sims_catutils')


(Scott Daniel) The PAL routine names are not the most understandable text ever written (I think they hail from the days when you had to worry about the length of your routine name).  I agree that we should replace the innards of the observationMetadataUtils.py methods with calls to PAL, but keeping those methods around as intelligible wrappers is probably a good idea.  I will open a new JIRA issue to this extent (and if you still think we should use the naked PAL, that issue can morph into replacing all of the instances of things like altAzToRaDec with calls to the naked PAL)


(Scott Daniel) I have moved it there.  I will open a separate pull request in sims_catalogs_generation just so you can make sure I didn't shoehorn it in inappropriately.


(Simon Krughoff) Your idea is exactly what I had in mind.  I don't just want the calls inside altAzToRaDec to be replaced.  I don't think users should ever have to call PAL by hand.


(Scott Daniel) Feature/SIM-90 use palpy
oops.  One more branch associated with this pull request

updated phoSimControlCatalog


(Scott Daniel) Feature/CATSIM-147 generalize findchipname
This branch adds an example script examples/findChipNameExample.py which shows how to find the chips associated with RAs and Decs on the LSST camera.

This is just a user-end implementation.  The code behind all of this is in the branch of the same name on sims_coordUtils.

Lynne: let me know if this is the kind of API you want for MAF.

* added example script that runs findChipName using an ObservationMetaData
generated from an actual OpSim pointing
* added explanatory comment to findChipNameExample.py


(Lynne Jones) If the ra/dec is not on the camera, is chipName just None?


(Scott Daniel) Yes.  findChipName is defined in Astrometry.py in sims_coordUtils if you want to see its inner workings.


(Lynne Jones) I might be confused here. You imported an 'Opsim3_61DBObject' but it's unused. Is that because it's not necessary, or is there a step missing from the example code?
Oh, okay, I went and looked more at the CatalogDBObject code and I see how this works. :) This implicitly means that OpsimDB is an opsim3_61DBObject.
So, looking at this further, I guess what we'd want to do is NOT use the opsim3_61DBObject (where I think 'getObservationMetaData' actually lives), but we'd want to use our own connection to the opsim database (there are some things about the Opsim3_61DbObject which do not hold true for new opsim database).

But, then I looked at the ObservationMetaData object and I see we can build one without having to go through the Opsim3_61DbObject anyway (we should just add a method to our OpsimDatabase class to make things easier). It's a little bit of code duplication, but probably worthwhile to avoid needing the whole catUtils package as well.


(Lynne Jones) Looks good.


(Scott Daniel) Yes, you should probably use your own code.  I did what I did to make sure I got a self-consistent ObservationMetaData object (where the RA, Dec are actually visible at the time of the MJD, etc.).  Any scheme to do so will work, as long as the ObservationMetaData object contains an RA, Dec, mjd, Site object, and rotTelPos.


(Simon Krughoff) I think the preferred way to get the root directory for a product this is eups.product_dir('obs_lsstSim')


(Simon Krughoff) Is there any reason not to do:
from lsst.obs.lsstSim import LsstSimMapper
mapper = LsstSimMapper()
return mapper.camera
instead of doing it by hand?


(Scott Daniel) No reason.  I did not know that functionality existed.  Is DM storing documentation anywhere other than the doxygen-generated html that comes with the code?


(Simon Krughoff) O.K.  I thought you might have been doing it for instructive purposes.  I'm sorry about the lack of documentation.  I'll try to find some, or add it if it doesn't exist.


(Scott Daniel) Bugfix/CATSIM-142 add all to modules
This branch adds the __all__= [] formalism to the python source code

added __all__ to sims_catUtils code


(Scott Daniel) Feature/SIM-1046 refactor photometric uncertainty
* fixed testGetters to reflect ObservationMetaData changes
* had to fix testObservationMetaDataGenerator.py to reflect
new ObservationMetaData m5 syntax
* did away with 'uRecalc' naming convention for galaxy magnitudes


(Scott Daniel) Feature/CATSIM-104 documentation and examples for
Most of the changes on this branch are new comments and docstrings I added to the example code to make it easier for users to follow what was being done.

I did add a new method to examples/catalogsExamples.py to search OpSim pointings by airmass.

In addition to the code changes here, I have a first draft of the catSim documentation done.

This page:

https://confluence.lsstcorp.org/display/SIM/Catalog+Simulations+Documentation

will link to all of the confluence pages documenting catSim.  There were some questions I could not address because I was unfamiliar with the code.  They can all be found on this page

https://confluence.lsstcorp.org/display/SIM/Catalog+Simulations+-+code+overview

They are:

What class of objects are stored in the Ebstars table?

What does the class fileDBObject in sims/catalogs/generation/db/ do? (does it work?)

What does validationFigures.py in sims_catUtils/examples/ do (when I tried to run it, I got an error)?

What are sersicUtils.py and validationUtils.py in sims/catUtils/ for?

* exampleAirMass in catalogsExamples.py now runs
* renamed output of exampleAirMass to make it a little more clear
* base airmass test on all stars
* added docstrings to the methods in catalogsExamples.py
* added docstring to GalaxyObj warning users that it does not cover the whole sky
* renamed 'Trim' methods in catalogsExamples.py
* added some comments to catalogsExamples.py
* added comment to generatePhosimInput.py
* added more comments to generatePhosimInput.py
* edited makeGalaxyPhotCat.py so that it runs more easily


(Simon Krughoff) ot --> to


(Simon Krughoff) Can't you do this using the utilities to make observation metadata from an alt, az pair?


(Simon Krughoff) I'm very sorry for taking so long on this review.  You'll have to be careful not to reintroduce the trim word again.


(Simon Krughoff) Why turn down the chunk size?  The bigger the chunk size, the faster it should go.


(Simon Krughoff) Can you add a comment like this to the GalaxyTileObject indicating that it is the one to use for all sky?

On second thought, I guess you already did that by commenting here.  Use your judgement.


(Scott Daniel) Yes... that wouldn't correspond to an actual Opsim pointing.  It would just have default values for Opsim_moonra, etc.  I'm not sure if that is important.  I will leave this as-is for now.


(Yusra AlSayyad) Bugfix/CATSIM-113
See https://jira.lsstcorp.org/browse/CATSIM-113


(Simon Krughoff) Is there anything blocking the merge of this branch?  Typically the workflow is to have the owner merge the branch when the review is approved and the review comments are addressed.


(Scott Daniel) Feature/CATSIM-157 catsim demos
Hi Simon, Bryce

I am drawing up tutorial scripts regarding CatSim for next week's user seminar.

If y'all have time (and only if you have time), would you mind looking at them and letting me know if there is either

a) anything you know about that is important that I have missed

b) anything you wonder about that is not covered

?

We won't be doing the CatSim tutorial until Thursday 11/20, so there's time.  I just wanted another set of eyes.

thanks,

Scott

* expanded tutorial02
* added some whitespace
* expanded tutorial03
* changed documentation
* renamed old tutorialScript03.py
* renamed the processed columns in tutorial04
* expanded tutorial04
* tutorial04 tells user to consult basCatalogModels
* tutorial01 now tells the user to consult exampleCatalogDefinitions
* fixed some typos


(Simon Krughoff) Sorry I didn't give input on this.  These look excellent.  Can you turn the ones with all the comments into iPython notebooks?  I think that would be extremely useful.


(Scott Daniel) and now you're going to make me learn iPython... sigh ;)

I'll get on that.  It will take a few days (I've seen iPython before; I haven't really used it in depth, though)


(Simon Krughoff) My hope is that you like it when you have a look.  You can feel free to merge these tutorials and play with iPython in your copious free time :)


I'm not sure we want to do this for 2015. I'm just about remove support for v0.1b explicitly force it to support OO APIs.


axe


Hmm again I don't want to fix this with a conditional for 2015 since I want to remove 0.1b from 2015.


See my note below.  This is not how keyed_output works.


I'm not sure why we want to branch this. Can you explain?


It looks like. It's all dead.


Right idea, but getting a thermodynamic variable is a two-step process.

Step 1) Update
Step 2) Call keyed_output

For instance, in python you would do:
``` python
HEOS = CoolProp.AbstractState('HEOS','Water')
HEOS.update(CoolProp.PQ_INPUTS, 101325, 0)
HEOS.keyed_output(CoolProp.iT) # or could be HEOS.T()
```


Will remove this in the next commit


I had no idea this existed.  Nifty.


for usage I'd expect `blank` to mean none


This looks correct to me. :shipit:


Calendar.prototype пиши вне function Calendar;
В функцию toHTML передай параметр id, и добавляй так: document.getElementByID(id).write('...') - так будет добавляться в определенное место.


Nice