CTDL JSON LD Ingest

The most important takeaway is the data can be programmatically output so once one credential is mapped and output, it’s repeatable across all credentials.  Data can be updated including more CTDL properties over time. 

This is a draft document that will be expanded upon based on questions and other input and will eventually be located on the Credential Engine technical site. 

Contact Jeanne Kitchens: jkitchens@credentialengine.org

Table of Contents

  1. Introduction
  2. What are all of the Credential Registry Publishing Options?
  3. How does a publisher decide if the API publishing or CTDL Ingest option is best for them?
  4. What does using the CTDL JSON LD Ingest System look like?
  5. What CTDL data is initially supported by the CTDL JSON LD Ingest System?
  6. What CTDL classes could your organization publish initially and test in the Credential Registry Sandbox?
  7. What would your organization need to do?
  8. How would Credential Engine help?
  9. How does CE schedule the ingest?
  10. What if there are errors?
  11. Resources

Introduction

What are all of the Credential Registry Publishing Options?

All Credential Registry publishing options require an approved Credential Registry account.  Publishing options offer an array of options intended to meet organizations where their current technical capabilities are and support moving towards longer-term options that go from primarily content expertise roles to technical expertise roles.

The publishing options requiring technical expertise are repeatable and scalable.  Thereby, once implemented they can automatically run.  For all technical options, detailed guidance and technical assistance is available. The Credential Engine technical site https://credreg.net makes guidance transparent and our technical team is accessible to provide guidance via meetings and other communications.

Options and Tools Description Publisher Role Primarily Content or Technical Applicability
Account Creation API API for creating Credential Registry accounts.  Requires technical support with the publisher’s team with experience working with APIs.  Technical 
Badge Publisher Tool Imports Open Badge information from badging platforms to publish to the Credential Registry  Requires person(s) familiar with and access to the publishers badges on platforms following the Open Badge specification.  Content
Bulk Upload  Provides options to select CTDL terms to include with a CSV template, instructions and samples.  Users download a template, populate it with information and upload it to publish to the Credential Registry   Requires person(s) familiar with and access to information needed to fill pre-defined CSV templates.   Content
Manual Entry Intended for small quantities, manually enter information in the Credential Registry Publishing System.  Requires person(s) familiar with and access to information needed to manually fill in pre-defined templates.  Content
Pathway Builder Interactive tool for building education and career pathways. The Pathway Builder offers a user-friendly drag and drop interface that allows you to visually construct your pathway. By using connectors, you can link the  Requires person(s) knowledgeable about the components of new or proposed education and career pathways. Content
Registry Publishing Assistant API API for publishing and maintaining currency of structured data from a database to the Credential Registry.  Publishers push data via the API to the Registry. Store CTIDs.

 

Technical

How does a publisher decide if the API publishing or CTDL Ingest option is best for them?

Both options require the publishing organization to have technical staff participation to set up a repeatable process.  For both options the initial process of mapping between structured data from the source system to the API or to output raw CTDL JSON LD data is essential.  The primary difference between the two options is with API publishing, the publisher automates pushing data via the Registry Assistant API to the Registry versus with CTDL JSON LD publishing, outputting raw CTDL data for the Credential Registry Ingest System to ingest and publish.

The decision is ultimately up to the publisher based on their technical capabilities and preferences. The Credential Engine technical team always recommends scheduling discussions with technical team members  and decision makers to weigh options.

What does using the CTDL JSON LD Ingest System look like?

  • An organization publishes data on their own website following the guidance in the documentation, as CTDL-formatted linked open data.
  • An automated process, managed either by the organization or by CE staff periodically fetches the data and updates the supported referenced resources in the Registry. 
    • Data Source: Output CTDL JSON LD (publish at URLs on your website) > Credential Engine: Ingest > Credential Engine: Publish to the Registry
  • If document formatting or processing errors occur, notifications are sent to the organization managing the ingestion process, so they can work with the publisher of the data to correct the error in the data that triggered the import error.

What CTDL data is initially supported by the CTDL JSON LD Ingest System?

What CTDL classes could your organization publish initially and test in the Credential Registry Sandbox?

What would your organization need to do?

How would Credential Engine help?

  • CE technical team can assist with mapping your structured data to the CTDL for the purpose of direct ingestion.
  • CE technical team can assist with providing an example CTDL JSON LD resource that your tech team can use to programmatically output resources to be ingested. 
  • CE technical team can schedule the ingest and work with your team to troubleshoot any issues in the Credential Registry sandbox.

How does CE schedule the ingest?

  • Implementation of a “workflow” identifying the published URLs in a GitHub repository: credential-registry-ingest-scheduler
  • The workflow should invoke the “credentialengine/credential-registry-publish-action@main” GitHub Action with inputs for urls to ingest, registry organization ID, API key, and registry environment.
  • Workflow triggering can be manual, scheduled, or based on changes to the configuration (via pull request or commit).

What if there are errors?

  • Errors will appear in GitHub Actions outputs. Notifications/Alerts can be configured. 
  • When an error occurs that blocks the completion of an ingestion task, communicate with the author of the data to get the data fixed and the task rerun (or wait for it to automatically rerun)

Resources