<div dir="ltr"><div>Hi Jonathan,</div><div><br></div>Just to chime in here a bit from the DataONE side of things... we already support several features which are of relevance to what you are discussing. We support "obsoletes/obsoletedBy" properties in the system metadata, and these provide a direct version chain indicating when one identifier represents a newer version replacing an older identifier. As soon as an EML document has been replaced by a newer version, the older versions no longer show up in DataONE search results. But they are still accessible if someone knows the identifier directly (e.g., via a citation). If someone accesses an older version directly, the top of the page prominently indicates that a newer version is available (for example, see <a href="https://search.dataone.org/#view/knb-lter-bnz.69.13">https://search.dataone.org/#view/knb-lter-bnz.69.13</a>). <div><br></div><div>in addition, for more complex rearrangements of data (for example, when several data packages get merged into one), we provide a mechanism for indicating that the new data set was derived from the multiple earlier data sets (using the prov:wasDerivedFrom property). This goes into the data package description. These complex derivation relationships now show up on the DataONE web site, showing the provenance relationships among objects directly. When creating these newly derived products, if you don't want the old packages to also show up in searches, then the old packages can be marked as 'archived'. When you do that, the new packages will show up in searches, and the old packages will be listed as the source of the new package, but only the new packages would show up in search results.<div><br></div><div>All of these features are available to the PASTA system when it submits metadata records to DataONE. I think it covers a lot of what you are requesting in this thread. Happy to help with any followup discussion where needed.</div><div><br></div><div>Matt<br><div><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Feb 28, 2016 at 11:32 AM, Margaret O'Brien <span dir="ltr"><<a href="mailto:margaret.obrien@ucsb.edu" target="_blank">margaret.obrien@ucsb.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey folks -<br>
These are issues of "Dataset design best practices", and we do have a working group for this. I think that pretty quickly, a group of 4-5 of us on VTC could iron out some recommendations for this particular question, based on James's email to Jim.<br>
<br>
Maybe we can schedule the April water cooler for this? My calendar says that the March subject is to continue with the IMC-NIMO relations. If anyone needs to talk over ideas ahead of that, we can schedule something ad hoc.<br>
<br>
Keep in mind that the removal of redundant datasets in D1 that Linda refers to was about the older datasets that came in from the Metacat node. Some sites did redesign their collections with the move to pasta, but these are really two different issues.<br>
<br>
Margaret<br>
<br>
-----------<br>
Margaret O'Brien<br>
Information Management<br>
Santa Barbara Coastal LTER<br>
Marine Science Institute, UCSB<br>
Santa Barbara, CA 93106<br>
<a href="tel:805-893-2071" value="+18058932071" target="_blank">805-893-2071</a> (voice)<br>
<a href="http://sbc.lternet.edu" rel="noreferrer" target="_blank">http://sbc.lternet.edu</a><span class=""><br>
<br>
On 2/28/16 10:16 AM, Jonathan Walsh wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
James makes a good point that I forgot to consider. Those stream temperature DOIs are already a part of the public domain and may well have been used and cited so it's silly to consider deleting them.<br>
<br>
It would be nice if we could control what PASTA lists for the search results so that deprecated data sets would still be available but not show up as a first choice for the simple search. And then maybe in advanced search there could be a toggle to display deprecated datasets. My goal is to keep our premiere datasets, which are more and more multi dataset packages, from being buried by our older, not-as-useful datasets to someone browsing our data.<br>
<br>
I like the idea of including provenance EML for each of the deprecated packages in my new multi dataset packages to provide a path forward and backward between the old and new. I would definitely like to do that. I'm also working on a multi dataset package for our telephone survey GIS componentry and it's a similar situation.<br>
<br>
A strategy to help ensure no new copies of deprecated packages get released into the wild would indeed be a good topic for a call.<br>
<br>
Thanks for a<br>
<br>
<br>
<br></span><span class="">
On Sat, Feb 27, 2016 at 4:52 PM, James Laundre <<a href="mailto:jlaundre@mbl.edu" target="_blank">jlaundre@mbl.edu</a> <mailto:<a href="mailto:jlaundre@mbl.edu" target="_blank">jlaundre@mbl.edu</a>>> wrote:<br>
<br>
Hi Jonathan,<br>
<br>
I have emailed Mark and Jame Brunt about deleting files from the<br>
LTER Network Data Portal since we are combining some of our yearly<br>
files into multiyear files. The email from James is below. I have<br>
just started the process of deprecating the old data sets and have<br>
not yet look into including the provenance EML.<br>
<br>
One suggestion I have is to put a note in the abstract of the<br>
deprecated data set that explains and points to the new multiyear<br>
data set. The abstract most likely will be read by people.<br>
<br>
Cheers,<br>
<br>
Jim<br>
<br>
<br></span>
*From: *"James Brunt" <<a href="mailto:jbrunt@lternet.edu" target="_blank">jbrunt@lternet.edu</a> <mailto:<a href="mailto:jbrunt@lternet.edu" target="_blank">jbrunt@lternet.edu</a>>><br>
*To: *<a href="mailto:jlaundre@mbl.edu" target="_blank">jlaundre@mbl.edu</a> <mailto:<a href="mailto:jlaundre@mbl.edu" target="_blank">jlaundre@mbl.edu</a>><br>
*Cc: *"Mark Servilla" <servilla@LTERnet.edu><br>
*Sent: *Monday, April 13, 2015 4:14:01 PM<br>
*Subject: *deleting data sets<div><div class="h5"><br>
<br>
-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
Hi Jim -<br>
<br>
Mark forwarded me your request regarding deleting data packages. If I<br>
understand correctly what you are doing is creating new data packages<br>
that cover a series of years that you want to replace the individual<br>
annual data packages.<br>
<br>
The individual packages that have already been published are basically<br>
in the public domain having been registered with DataCite and received<br>
a DOI, and been contributed to DataONE and from there possibly beyond.<br>
These data packages have potentially been used and cited in journals<br>
and we have an obligation to make sure the original is still<br>
available.<br>
<br>
We can however make sure that no new copies of the deprecated data<br>
packages get released into the wild. This might require a call to<br>
discuss further but basically the process would be to update all of<br>
the impacted data packages with a revison that closes public read<br>
access in the EML to make them private. I'm assuming that you would<br>
issue new ID numbers. (If you were planning to update say the first<br>
ID number in each series this would still work to deprecate all the<br>
other data package IDs.)<br>
<br>
If you wanted to make a slightly cleaner and more elegant transition<br>
you could include the provenance EML for each of the deprecated<br>
packages in your new package EML. That way there is a path forward and<br>
backward between the old and the new.<br>
<br>
I'm sure this probably isn't what you want to hear since you were<br>
probably hoping to create a more compact list of ARC data packages.<br>
There were a number of finer grained tweaks to control the display<br>
like this that we had hoped to implement that had to be abandoned when<br>
NSF cut our funding.<br>
<br>
All that said, it is still technically possible to delete a data<br>
package from PASTA but it's only through the API and wouldn't have any<br>
effect on those records already in DataONE and the wild. The Scope and<br>
Identifier are marked as deleted and cannot subsequently be reused. We<br>
would discourage this for the reason of our public obligation stated<br>
above and have only used it under extreme circumstances.<br>
<br>
I'm happy to continue this discussion to fine tune your strategy as<br>
you feel necessary.<br>
<br>
Regards,<br>
<br>
James<br>
<br>
<br></div></div>
------------------------------------------------------------------------<br>
*From: *"Linda A Powell" <<a href="mailto:powell@fiu.edu" target="_blank">powell@fiu.edu</a> <mailto:<a href="mailto:powell@fiu.edu" target="_blank">powell@fiu.edu</a>>><br>
*To: *"Jonathan Walsh" <<a href="mailto:walshjcaryinstitute@gmail.com" target="_blank">walshjcaryinstitute@gmail.com</a><br>
<mailto:<a href="mailto:walshjcaryinstitute@gmail.com" target="_blank">walshjcaryinstitute@gmail.com</a>>>, "IM committee"<br>
<<a href="mailto:im@lternet.edu" target="_blank">im@lternet.edu</a> <mailto:<a href="mailto:im@lternet.edu" target="_blank">im@lternet.edu</a>>><br>
*Sent: *Saturday, February 27, 2016 1:02:36 PM<br>
*Subject: *Re: [LTER-im] Removing from PASTA?<span class=""><br>
<br>
<br>
Hi Jonathan,<br>
<br>
<br>
A short while ago the entire IM community with the exception of<br>
Suzanne (via doodle poll) wanted to have all the old metacat<br>
(PASTA) records (that we thought were deleted) removed from being<br>
seen in DataOne. Suzanne agreed with this practice but can't have<br>
her files deleted yet as they are still transferring files into<br>
PASTA. Mark was going to write a script for DataOne that would<br>
hide/remove these old files from the DataOne users and I don't<br>
know where he is in the process but hopefully it will be finished<br>
soon.<br>
<br>
<br>
I've not deleted a file in PASTA yet so I don't know how well the<br>
process works. I suspect that the old, removed, files might not<br>
show up in PASTA but may be pushed to DataOne. Hopefully Mark can<br>
speak to this.<br>
<br>
<br>
Best,<br>
<br>
<br>
Linda<br>
<br>
<br>
Linda Powell<br>
Information Manager<br>
Florida Coastal Everglades LTER Program<br>
OE 148, Florida International University<br>
University Park<br>
Miami, Florida 33199<br></span>
Phone (Tallahassee, FL): <a href="tel:850-745-0381" value="+18507450381" target="_blank">850-745-0381</a> <tel:<a href="tel:850-745-0381" value="+18507450381" target="_blank">850-745-0381</a>><br>
Phone(Miami,FL): <a href="tel:305-856-0039" value="+13058560039" target="_blank">305-856-0039</a> <tel:<a href="tel:305-856-0039" value="+13058560039" target="_blank">305-856-0039</a>> or <a href="tel:305-348-6054" value="+13053486054" target="_blank">305-348-6054</a><br>
<tel:<a href="tel:305-348-6054" value="+13053486054" target="_blank">305-348-6054</a>><br>
Website: <a href="http://fcelter.fiu.edu" rel="noreferrer" target="_blank">http://fcelter.fiu.edu</a><br>
<br>
<br>
------------------------------------------------------------------------<br>
*From:* im <<a href="mailto:im-bounces@lists.lternet.edu" target="_blank">im-bounces@lists.lternet.edu</a><br>
<mailto:<a href="mailto:im-bounces@lists.lternet.edu" target="_blank">im-bounces@lists.lternet.edu</a>>> on behalf of Jonathan Walsh<br>
<<a href="mailto:walshjcaryinstitute@gmail.com" target="_blank">walshjcaryinstitute@gmail.com</a> <mailto:<a href="mailto:walshjcaryinstitute@gmail.com" target="_blank">walshjcaryinstitute@gmail.com</a>>><br>
*Sent:* Saturday, February 27, 2016 12:05 PM<br>
*To:* IM committee<br>
*Subject:* [LTER-im] Removing from PASTA?<span class=""><br>
I think this topic has come up in the past but I do not recall the<br>
outcome and if so I apologize for that.<br>
<br>
How can I remove records from PASTA? I have a whole bunch of<br>
stream temperature files that I would like to combine. Then I<br>
would like to remove the old ones.<br>
<br>
The reason for this is when one browses BES on PASTA one sees<br>
mostly stream temperature files and it's confusing.<br>
<br>
Thank you<br>
<br>
<br>
<br>
-- Information Manager, Baltimore Ecosystem Study<br>
Institute of Ecosystem Studies<br>
Box AB; Route 44A<br>
Millbrook, NY 12545-0129<br></span>
P: <a href="tel:845%2F677%2F7600%20Extension%20103" value="+18456777600" target="_blank">845/677/7600 Extension 103</a> <tel:845%2F677%2F7600%20Extension%20103><br>
F: <a href="tel:845%2F677%2F5976" value="+18456775976" target="_blank">845/677/5976</a> <tel:845%2F677%2F5976><span class=""><br>
E: WalshJ@EcoStudies.org<br>
<br>
_______________________________________________<br>
Long Term Ecological Research Network<br>
im mailing list<br></span>
<a href="mailto:im@lternet.edu" target="_blank">im@lternet.edu</a> <mailto:<a href="mailto:im@lternet.edu" target="_blank">im@lternet.edu</a>><span class=""><br>
<br>
<br>
<br>
<br>
<br>
-- <br>
Information Manager, Baltimore Ecosystem Study<br>
Institute of Ecosystem Studies<br>
Box AB; Route 44A<br>
Millbrook, NY 12545-0129<br></span>
P: <a href="tel:845%2F677%2F7600%20Extension%20103" value="+18456777600" target="_blank">845/677/7600 Extension 103</a> <tel:845%2F677%2F7600%20Extension%20103><br>
F: <a href="tel:845%2F677%2F5976" value="+18456775976" target="_blank">845/677/5976</a> <tel:845%2F677%2F5976><span class=""><br>
E: WalshJ@EcoStudies.org<br>
<br>
<br>
_______________________________________________<br>
Long Term Ecological Research Network<br>
im mailing list<br>
<a href="mailto:im@lternet.edu" target="_blank">im@lternet.edu</a><br>
<br>
</span></blockquote><div class="HOEnZb"><div class="h5">
<br>
_______________________________________________<br>
Long Term Ecological Research Network<br>
im mailing list<br>
<a href="mailto:im@lternet.edu" target="_blank">im@lternet.edu</a><br>
<br>
</div></div></blockquote></div><br></div>