Extension/index page
From LinkedGov
Contents |
Overview
Refine's "index" page is home to 3 panels
- Create new project
- Open existing project
- Import a project
"Create new project" - allows the user to upload a file or specify a web URL to import a file from in various formats (CSV, TSV, XLS, RDF, XML…)
"Open existing project" - allows the user to open a project they have already created (the list of projects are stored within Refine's "app data" folder on the machine).
"Import a project" - lets the user import an exported Refine project (in the form of a .zip file).
The LinkedGov extension will make use of the first two panels - so they can create a new project or resume a project they have already started.
Note: the three panels have a navigation menu on the left hand side of the screen, but our extension removes this menu so the user isn't able to access the "import project" panel.
Import/upload new data
The first page the user lands on (and when Google Refine is first run) is the "Upload your data" page, or internally in Refine, the "index" page.
From here, the user can upload a data file from their machine or enter a web address where the data is located online.
Saving metadata
Once the user has filled out our metadata form when importing their data and clicked "Next", a JSON object is constructed as the form is validated.
The JSON object is in key-value form, with the following keys to save the dataset's metadata:
- LinkedGov.name
- LinkedGov.license
- LinkedGov.licenseLocation
- LinkedGov.frequency
- LinkedGov.organisation
- LinkedGov.datePublished
- LinkedGov.source
- LinkedGov.webLocation
- LinkedGov.descriptionLocation
- LinkedGov.keywords
JSON object:
var metadataObject = {
"LinkedGov.name":"Allotment locations in Trafford",
"LinkedGov.license":"Open Government License",
"LinkedGov.licenseLocation":"http://www.nationalarchives.gov.uk/doc/open-government-licence/",
"LinkedGov.frequency":"Never",
"LinkedGov.organisation":"Trafford Council",
"LinkedGov.datePublished":"2011-03-15",
"LinkedGov.source":"http://www.trafford.gov.uk/opendata/sets/allotments.xml",
"LinkedGov.webLocation":"http://www.trafford.gov.uk/opendata/sets/allotments.xml",
"LinkedGov.descriptionLocation":"http://www.trafford.gov.uk/opendata",
"LinkedGov.keywords":"allotments, location, latitude, longitude, trafford"
};
Once the object has been built, we need to store these values so we can access them on the next page - allowing us to save the metadata in RDF along with the data. We can do this using Refine's 'set-preference' and 'get-preference' commands - which allows us to save a key and it's value.
NOTE - these commands require us to pass the project ID as a parameter with each "set" and "get-preference" call.
As the project ID is created at the very last moment of the importing stage, the only way to access the project ID is by overriding one of Refine's functions - "pollImportJob".
"pollImportJob", periodically checks whether the data has been successfully imported. If it has been successfully imported, the project ID is finally returned inside a JSON object called "job" - which we can intercept:
/*
* Store the original 'pollImportJob' before we overwrite Refine's version.
*/
LinkedGov.pollImportJob = Refine.CreateProjectUI.prototype.pollImportJob;
Refine.CreateProjectUI.prototype.pollImportJob = function(start, jobID, timerID, checkDone, callback, onError) {
/*
* Create our own "callback" function
*/
lgCallback = function(jobID,job) {
/*
* This function is accessed twice by Refine, the first time to
* send the user to the "preview" panel when they can modify the
* import options, and the second time, with the projectID to the
* "project" page.
*
* The second time round is when the projectID is present, so we
* intercept it to make a call to save our custom metadata.
*/
if(typeof job.config.projectID != 'undefined'){
LinkedGov.saveMetadata(jobID, job.config.projectID, function(jobID, projectID){
Refine.CreateProjectUI.cancelImportinJob(jobID);
document.location = "project?project=" + projectID;
});
} else {
callback(jobID,job);
}
};
/*
* Call the original 'pollImportJob' function that was stored at the end of our new
* 'pollImportJob' function.
*/
LinkedGov.pollImportJob(start, jobID, timerID, checkDone, lgCallback, onError);
}
Once we have intercepted the "projectID", we pass it to our "saveMetadata" function, which makes a "set-preference" call for each of our keys to store their value:
/*
* saveMetadata
*
* Posts the metadata object to Refine's "set-preference" command
* which stores the key-value pairs in the project's metadata.json file.
*/
saveMetadata:function(jobID, projectID, callback){
var self = this;
$.each(LinkedGov.vars.metadataObject,function(key,val){
$.ajax({
type: "POST",
url: "/command/core/set-preference?" + $.param({
name: key,
value : encodeURIComponent(val),
project : projectID
}),
dataType: "json"
});
});
callback(jobID,projectID);
},
Once each "set-preference" call has been made, we then let Refine redirect the user to the project page as normal using the project ID - "/project?projectID=123456789".
Configure parsing options
After successfully filling out a metadata form they can proceed to the next panel, the importing ('parsing' in Refine) options, where they can modify the way their data has been imported.
Changes
- The parsing options stage is hidden by default as it contains quite a few off-putting options such as character encodings and delimiters. So the user has to make the choice to correct the import.
- The parsing options have been converted into a collection of expanding questions that open up to reveal the parsing options, we've put a number of expandable panels on the page that breaks the parsing options into easy-to-understand sections, asking the user questions:
- about column and row numbers, about separation and about type detection.
Resume import
The "open existing project" panel displays a list of created projects local to the machine with their name, date modified and the ability to delete and rename them.
Refine stores all created projects in an "app data" folder (which differs in name and location depending on the OS).
Each project has it's own folder, which contains
- metadata.json
- data.zip
- /history
The /history folder contains a zip file of the project data at each action of the undo/redo history.
Displaying additional metadata
It's possible for us to display our custom metadata that we collect during import stage alongside each project when resuming the project, but we would need to inject the new columns of data into the table - either by overridding Refine's project-list creation function or by injecting in dynamically.
UI notes
The "create new project" panel is built in a way that makes it hard to restyle the page exactly how we want it - the use of nested tables, Javascript resizing and modular panels means we can only inject our metadata form into the bottom half of the table.
Importing issues
There are issues with some types of data when importing.
